[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:00 dist-test-slave-h6n5 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:00 dist-test-slave-h6n5 krb5kdc[399](info): set up 2 sockets
Jun 20 14:19:00 dist-test-slave-h6n5 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:19:01.724121 392 mini_kdc.cc:121] Time spent starting KDC: real 1.987s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:01 dist-test-slave-h6n5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:01.842859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:01.931890 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:01.932125 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:01.932164 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:01.933359 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:01.933434 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:01.933461 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:01.933509 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:01.933540 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:01.935101 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:01.935382 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:01.935689 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:01.940765 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:01.940843 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:01.940789 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:01.940975 412 server_base.cc:1034] running on GCE node
I20260620 14:19:01.941254 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:01.941560 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:01.942734 412 hybrid_clock.cc:648] HybridClock initialized: now 1781965141942721 us; error 54 us; skew 500 ppm
Jun 20 14:19:01 dist-test-slave-h6n5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:01.948788 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:01.949432 412 webserver.cc:460] Webserver started at http://127.0.98.62:37125/ using document root <none> and password file <none>
I20260620 14:19:01.949640 412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:01.949684 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:01.949781 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:01.950937 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b"
format_stamp: "Formatted at 2026-06-20 14:19:01 on dist-test-slave-h6n5"
server_key: "1d61c0ff134dfcf5e0e2837fef33d22a"
server_key_iv: "ffa9a0d70a7f6bc7dded26f19e98210e"
server_key_version: "encryptionkey@0"
I20260620 14:19:01.951256 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b"
format_stamp: "Formatted at 2026-06-20 14:19:01 on dist-test-slave-h6n5"
server_key: "1d61c0ff134dfcf5e0e2837fef33d22a"
server_key_iv: "ffa9a0d70a7f6bc7dded26f19e98210e"
server_key_version: "encryptionkey@0"
I20260620 14:19:01.952888 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:01.953656 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:01.953850 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:01.953941 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b"
format_stamp: "Formatted at 2026-06-20 14:19:01 on dist-test-slave-h6n5"
server_key: "1d61c0ff134dfcf5e0e2837fef33d22a"
server_key_iv: "ffa9a0d70a7f6bc7dded26f19e98210e"
server_key_version: "encryptionkey@0"
I20260620 14:19:01.954000 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:01.973691 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:01.976171 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:01.976301 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:01.980410 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34381
I20260620 14:19:01.980482 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34381 every 8 connection(s)
I20260620 14:19:01.980857 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:01.981827 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:01.984304 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b: Bootstrap starting.
I20260620 14:19:01.984913 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:01.985208 480 log.cc:824] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:01.985991 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b: No bootstrap required, opened a new log
I20260620 14:19:01.987543 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34381 } }
I20260620 14:19:01.987774 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:01.987797 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3d8ed203ac4caabe69e79d7cd2da1b, State: Initialized, Role: FOLLOWER
I20260620 14:19:01.987973 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34381 } }
I20260620 14:19:01.988065 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:01.988111 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:01.988165 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:01.988220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 412
I20260620 14:19:01.988341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:01.988801 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34381 } }
I20260620 14:19:01.988914 480 leader_election.cc:302] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc3d8ed203ac4caabe69e79d7cd2da1b; no voters:
I20260620 14:19:01.988996 392 external_mini_cluster.cc:1428] Setting key 374bead53967d6dfcac8a955c519f800
I20260620 14:19:01.989069 480 leader_election.cc:288] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:01.989241 484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:01.989360 484 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [term 1 LEADER]: Becoming Leader. State: Replica: dc3d8ed203ac4caabe69e79d7cd2da1b, State: Running, Role: LEADER
I20260620 14:19:01.989382 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:01.989444 484 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34381 } }
I20260620 14:19:01.989959 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34381 } } }
I20260620 14:19:01.990056 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:01.990200 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc3d8ed203ac4caabe69e79d7cd2da1b. Latest consensus state: current_term: 1 leader_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3d8ed203ac4caabe69e79d7cd2da1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34381 } } }
I20260620 14:19:01.990247 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b [sys.catalog]: This master's current role is: LEADER
W20260620 14:19:01.991235 502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:19:01.991307 502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:19:01.991379 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:01.991603 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:01.993039 493 catalog_manager.cc:1347] Generated new cluster ID: 0582b176035b4ed0812b6d171a7e410a
I20260620 14:19:01.993098 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:02.000180 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:02.000844 493 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 14:19:01 dist-test-slave-h6n5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:02.003890 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc3d8ed203ac4caabe69e79d7cd2da1b: Generated new TSK 0
I20260620 14:19:02.004050 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:02.007863 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:01.990369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.98.62:34381)
0620 14:19:01.990829 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:01.990838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:01.990958 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:19:01.993344 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:01.993351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:01.993404 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:01.993541 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:01.995616 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:01.997015 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:01.998620 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:01.999220 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:01.999323 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.004072 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:02.004077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:02.004078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:02.004090 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:02.006025 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.006435 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.006438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.006439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.006547 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.006786 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.006787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.006787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.007061 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:02.007193 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:02.007414 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 14:19:02.007587 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:02.034013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34381
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:02.128288 507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:02.128587 507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:02.128618 507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:02.129819 507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:02.129860 507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:02.129889 507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:02.131356 507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:02.131632 507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:02.131883 507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:02.136781 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:02.136904 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:02.137143 507 server_base.cc:1034] running on GCE node
W20260620 14:19:02.136781 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:02.137425 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:02.137717 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:02.138900 507 hybrid_clock.cc:648] HybridClock initialized: now 1781965142138883 us; error 31 us; skew 500 ppm
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:02.141430 507 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:02.141952 507 webserver.cc:460] Webserver started at http://127.0.98.1:35173/ using document root <none> and password file <none>
I20260620 14:19:02.142181 507 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:02.142225 507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:02.142366 507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:02.143410 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "42f55d7e91a0480187203b571fafc9cb"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "f6db4f7058478b142042f520aec7a08b"
server_key_iv: "f0012efd84c6f054cc92b26e62dc39ed"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.143785 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "42f55d7e91a0480187203b571fafc9cb"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "f6db4f7058478b142042f520aec7a08b"
server_key_iv: "f0012efd84c6f054cc92b26e62dc39ed"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.145085 507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:02.145776 522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.145943 507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:02.146037 507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "42f55d7e91a0480187203b571fafc9cb"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "f6db4f7058478b142042f520aec7a08b"
server_key_iv: "f0012efd84c6f054cc92b26e62dc39ed"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.146113 507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:02.161439 507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:02.163786 507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:02.163949 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:02.164175 507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:02.164501 507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:02.164559 507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.164599 507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:02.164628 507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.170315 507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34189
I20260620 14:19:02.170359 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34189 every 8 connection(s)
I20260620 14:19:02.170660 507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:02.179288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 507
I20260620 14:19:02.179378 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.171354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59731 (local address 127.0.98.62:34381)
0620 14:19:02.171402 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:02.171404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:02.171578 (+ 174us) server_negotiation.cc:407] Connection header received
0620 14:19:02.172071 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:02.172072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:02.172081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:02.172135 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:02.172996 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.173486 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.174421 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.174643 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.174699 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.176737 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:02.176738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:02.176738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:02.176745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:02.178077 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.178488 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.178489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.178489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.178540 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.178775 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.178776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.178776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.178893 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:02.178958 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:02.179219 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:19:02.179286 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:19:02.179388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:02.179594 392 external_mini_cluster.cc:1428] Setting key dcf1655a726da13e0a68df0a84ed8aa1
I20260620 14:19:02.179638 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.171303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34381 (local address 127.0.98.1:59731)
0620 14:19:02.171451 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:02.171490 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:02.171990 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:02.172192 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:02.172195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:02.172726 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:02.172913 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:02.172917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.173575 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.173576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:02.174340 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:02.174342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.174721 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.174721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:02.174814 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.175302 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:02.175310 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:02.176652 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:02.178153 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:02.178158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:02.178159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:02.178370 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:02.178597 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:02.178598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:02.178598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:02.178691 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:02.178958 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:02.178961 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:02.179064 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 14:19:02.179196 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 14:19:02.179358 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260620 14:19:02.180181 636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34381
I20260620 14:19:02.180294 636 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:02.180492 636 heartbeater.cc:511] Master 127.0.98.62:34381 requested a full tablet report, sending...
I20260620 14:19:02.181031 443 ts_manager.cc:194] Registered new tserver with Master: 42f55d7e91a0480187203b571fafc9cb (127.0.98.1:34189)
I20260620 14:19:02.181924 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59731
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:02.204692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34381
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:02.287604 641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:02.287812 641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:02.287835 641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:02.288966 641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:02.289006 641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:02.289033 641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:02.290611 641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:02.290799 641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:02.290962 641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:02.295950 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:02.295930 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:02.296087 641 server_base.cc:1034] running on GCE node
W20260620 14:19:02.295918 646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:02.296325 641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:02.296581 641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:02.297744 641 hybrid_clock.cc:648] HybridClock initialized: now 1781965142297729 us; error 34 us; skew 500 ppm
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:02.300196 641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:02.300650 641 webserver.cc:460] Webserver started at http://127.0.98.2:37893/ using document root <none> and password file <none>
I20260620 14:19:02.300871 641 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:02.300915 641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:02.301066 641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:02.302084 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "144a4c7d78b74529bacff033244a7c79"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "ef651db034bbae7d2c9dafca762cc6c9"
server_key_iv: "12321781a2ba9d7fd9001dbb6a531c19"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.302434 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "144a4c7d78b74529bacff033244a7c79"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "ef651db034bbae7d2c9dafca762cc6c9"
server_key_iv: "12321781a2ba9d7fd9001dbb6a531c19"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.303784 641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:19:02.304499 656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.304692 641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:02.304787 641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "144a4c7d78b74529bacff033244a7c79"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "ef651db034bbae7d2c9dafca762cc6c9"
server_key_iv: "12321781a2ba9d7fd9001dbb6a531c19"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.304867 641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:02.314203 641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:02.316534 641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:02.316694 641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:02.316917 641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:02.317206 641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:02.317260 641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.317301 641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:02.317327 641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.323248 641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45273
I20260620 14:19:02.323288 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45273 every 8 connection(s)
I20260620 14:19:02.323699 641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:02.329218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 641
I20260620 14:19:02.329301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:02.329488 392 external_mini_cluster.cc:1428] Setting key c54f379a1e91845706b785e05c06ece3
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:02.333985 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.324483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40745 (local address 127.0.98.62:34381)
0620 14:19:02.324535 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:02.324537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:02.324746 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:19:02.325196 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:02.325197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:02.325209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:02.325276 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:02.326175 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.326684 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.328113 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.328296 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.328350 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.331009 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:02.331011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:02.331011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:02.331017 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:02.332468 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.332959 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.332960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.332961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.333013 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.333311 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.333312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.333312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.333451 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:02.333508 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:02.333863 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 14:19:02.333924 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:19:02.334249 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.324410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34381 (local address 127.0.98.2:40745)
0620 14:19:02.324592 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:02.324633 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:02.325114 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:02.325332 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:02.325335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:02.325887 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:02.326080 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:02.326085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.326787 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.326788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:02.328035 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:02.328037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.328383 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.328384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:02.328507 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.329059 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:02.329069 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:02.330911 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:02.332564 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:02.332569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:02.332572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:02.332884 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:02.333076 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:02.333077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:02.333078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:02.333238 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:02.333530 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:02.333538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:02.333713 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:19:02.333840 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:19:02.334014 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":117,"threads_started":1}
I20260620 14:19:02.334789 770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34381
I20260620 14:19:02.334873 770 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:02.335065 770 heartbeater.cc:511] Master 127.0.98.62:34381 requested a full tablet report, sending...
I20260620 14:19:02.335469 443 ts_manager.cc:194] Registered new tserver with Master: 144a4c7d78b74529bacff033244a7c79 (127.0.98.2:45273)
I20260620 14:19:02.335875 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40745
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:02.354585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34381
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:02.435480 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:02.435757 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:02.435794 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:02.436966 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:02.437047 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:02.437098 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:02.438939 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:02.439208 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:02.439436 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:02.444722 775 server_base.cc:1034] running on GCE node
W20260620 14:19:02.444687 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:02.444784 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:02.444684 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:02.445138 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:02.445380 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:02.446524 775 hybrid_clock.cc:648] HybridClock initialized: now 1781965142446505 us; error 32 us; skew 500 ppm
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:02.449055 775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:02.449528 775 webserver.cc:460] Webserver started at http://127.0.98.3:44473/ using document root <none> and password file <none>
I20260620 14:19:02.449749 775 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:02.449795 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:02.449924 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:02.451009 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "f57183e7f4c84a53bd1fc08261a38a2c"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "160eeb77ec235006970f4ec3b0a09c15"
server_key_iv: "7d654d8159a6bbc1de3d8f1de8023c2d"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.451366 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f57183e7f4c84a53bd1fc08261a38a2c"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "160eeb77ec235006970f4ec3b0a09c15"
server_key_iv: "7d654d8159a6bbc1de3d8f1de8023c2d"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.452713 775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:02.453509 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.453706 775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.453804 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "f57183e7f4c84a53bd1fc08261a38a2c"
format_stamp: "Formatted at 2026-06-20 14:19:02 on dist-test-slave-h6n5"
server_key: "160eeb77ec235006970f4ec3b0a09c15"
server_key_iv: "7d654d8159a6bbc1de3d8f1de8023c2d"
server_key_version: "encryptionkey@0"
I20260620 14:19:02.453869 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:02.465305 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:02.467242 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:02.467399 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:02.467689 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:02.467990 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:02.468021 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.468075 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:02.468103 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:02.474423 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40249
I20260620 14:19:02.474469 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40249 every 8 connection(s)
I20260620 14:19:02.474784 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:02.479271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 775
I20260620 14:19:02.479374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:02.479525 392 external_mini_cluster.cc:1428] Setting key 3c24c15dc6097a2cbd2564e99a8ab63f
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:02.484702 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.475956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36663 (local address 127.0.98.62:34381)
0620 14:19:02.476034 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:02.476036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:02.476228 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:19:02.476748 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:02.476749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:02.476758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:02.476812 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:02.477790 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.478287 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.479282 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.479620 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.479689 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.482079 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:02.482080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:02.482081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:02.482089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:02.483258 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.483741 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.483742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.483742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.483803 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.484056 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.484057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.484057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.484180 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:02.484257 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:02.484528 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:19:02.484615 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:19:02.484879 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.475943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34381 (local address 127.0.98.3:36663)
0620 14:19:02.476106 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:02.476147 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:02.476665 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:02.476863 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:02.476866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:02.477511 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:02.477704 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:02.477709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.478374 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.478374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:02.479177 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:02.479179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.479697 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.479697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:02.479796 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.480511 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:02.480521 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:02.481979 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:02.483354 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:02.483360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:02.483363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:02.483658 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:02.483867 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:02.483868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:02.483869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:02.483981 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:02.484247 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:02.484253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:02.484393 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 14:19:02.484499 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:19:02.484648 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260620 14:19:02.485342 904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34381
I20260620 14:19:02.485437 904 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:02.485613 904 heartbeater.cc:511] Master 127.0.98.62:34381 requested a full tablet report, sending...
I20260620 14:19:02.486014 443 ts_manager.cc:194] Registered new tserver with Master: f57183e7f4c84a53bd1fc08261a38a2c (127.0.98.3:40249)
I20260620 14:19:02.486384 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36663
I20260620 14:19:02.491619 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:02.514170 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.506566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.0.98.1:34189)
0620 14:19:02.506732 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:02.506737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:02.506748 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:02.506774 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:02.506776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:02.506789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:02.506866 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:02.507412 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.508255 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.509039 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.509350 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.509407 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.511586 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:02.511599 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:02.511599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:02.511606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:02.513204 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.513497 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.513499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.513500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.513551 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.513692 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.513692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.513692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.513801 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:02.513895 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:02.513960 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:19:02.514019 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
W20260620 14:19:02.514536 550 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38782
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:02.537019 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:02.529826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.0.98.1:34189)
0620 14:19:02.529933 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:02.529935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:02.529963 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:19:02.530067 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:02.530068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:02.530085 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:02.530153 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:02.530769 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.531260 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.532069 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:02.532272 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:02.532347 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:02.534575 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:02.534576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:02.534577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:02.534583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:02.535950 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.536267 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.536268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.536268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.536316 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:02.536484 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:02.536485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:02.536485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:02.536591 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:02.536679 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:02.536826 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:19:02.536911 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:19:02.537544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 507
I20260620 14:19:02.542742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 641
I20260620 14:19:02.548378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 775
I20260620 14:19:02.552997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 412
2026-06-20T14:19:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2842 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[918](info): set up 2 sockets
Jun 20 14:19:02 dist-test-slave-h6n5 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260620 14:19:03.595170 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:03 dist-test-slave-h6n5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:03.714156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:03.794960 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:03.795166 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:03.795189 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:03.796427 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:03.796473 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:03.796487 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:03.796500 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:03.796514 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:03.798049 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:03.798329 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:03.798754 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:03.804051 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:03.804040 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:03.804073 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:03.804272 931 server_base.cc:1034] running on GCE node
I20260620 14:19:03.804529 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:03.804837 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:03.806035 931 hybrid_clock.cc:648] HybridClock initialized: now 1781965143806021 us; error 27 us; skew 500 ppm
Jun 20 14:19:03 dist-test-slave-h6n5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:03.808600 931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:03.809072 931 webserver.cc:460] Webserver started at http://127.0.98.62:43889/ using document root <none> and password file <none>
I20260620 14:19:03.809324 931 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:03.809392 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:03.809535 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:03.810617 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "81afd470a8364dfdb3f2fb416bd6a389"
format_stamp: "Formatted at 2026-06-20 14:19:03 on dist-test-slave-h6n5"
server_key: "7df88c10e84b2d3955afa281b6ceb9d5"
server_key_iv: "58d99dda860fd5d31d2bded7c01ba20a"
server_key_version: "encryptionkey@0"
I20260620 14:19:03.810981 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "81afd470a8364dfdb3f2fb416bd6a389"
format_stamp: "Formatted at 2026-06-20 14:19:03 on dist-test-slave-h6n5"
server_key: "7df88c10e84b2d3955afa281b6ceb9d5"
server_key_iv: "58d99dda860fd5d31d2bded7c01ba20a"
server_key_version: "encryptionkey@0"
I20260620 14:19:03.812412 931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:03.813179 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:03.813362 931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:03.813453 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "81afd470a8364dfdb3f2fb416bd6a389"
format_stamp: "Formatted at 2026-06-20 14:19:03 on dist-test-slave-h6n5"
server_key: "7df88c10e84b2d3955afa281b6ceb9d5"
server_key_iv: "58d99dda860fd5d31d2bded7c01ba20a"
server_key_version: "encryptionkey@0"
I20260620 14:19:03.813535 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:03.836493 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:03.838989 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:03.839159 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:03.843273 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39053
I20260620 14:19:03.843325 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39053 every 8 connection(s)
I20260620 14:19:03.843698 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:03.844563 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:03.846709 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389: Bootstrap starting.
I20260620 14:19:03.847221 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:03.847467 999 log.cc:824] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:03.848101 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389: No bootstrap required, opened a new log
I20260620 14:19:03.849149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 931
I20260620 14:19:03.849246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:03.849402 392 external_mini_cluster.cc:1428] Setting key 57d2a63ac26107137f8588ab9ce493ff
I20260620 14:19:03.849910 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afd470a8364dfdb3f2fb416bd6a389" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260620 14:19:03.850106 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:03.850152 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81afd470a8364dfdb3f2fb416bd6a389, State: Initialized, Role: FOLLOWER
I20260620 14:19:03.850276 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afd470a8364dfdb3f2fb416bd6a389" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260620 14:19:03.850348 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:03.850382 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:03.850448 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:03.851447 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afd470a8364dfdb3f2fb416bd6a389" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260620 14:19:03.851578 999 leader_election.cc:302] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81afd470a8364dfdb3f2fb416bd6a389; no voters:
I20260620 14:19:03.851724 999 leader_election.cc:288] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:03.851907 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:03.851948 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:03.852062 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [term 1 LEADER]: Becoming Leader. State: Replica: 81afd470a8364dfdb3f2fb416bd6a389, State: Running, Role: LEADER
I20260620 14:19:03.852156 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afd470a8364dfdb3f2fb416bd6a389" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260620 14:19:03.852478 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81afd470a8364dfdb3f2fb416bd6a389. Latest consensus state: current_term: 1 leader_uuid: "81afd470a8364dfdb3f2fb416bd6a389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afd470a8364dfdb3f2fb416bd6a389" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } } }
I20260620 14:19:03.852577 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:03.852485 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81afd470a8364dfdb3f2fb416bd6a389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81afd470a8364dfdb3f2fb416bd6a389" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } } }
I20260620 14:19:03.852672 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:03.852770 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:03.853266 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:03.854847 1013 catalog_manager.cc:1347] Generated new cluster ID: 71f5871698c642e5b71944b661fd04fd
I20260620 14:19:03.854909 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:03 dist-test-slave-h6n5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:03.861052 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:03.849913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.0.98.62:39053)
0620 14:19:03.850101 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:03.850111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:03.850144 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:19:03.850257 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:03.850264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:03.850318 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:03.850465 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:03.851066 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:03.852523 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:03.853671 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:03.854266 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:03.854372 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:03.857140 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:03.857145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:03.857145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:03.857157 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:03.859223 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:03.859618 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:03.859622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:03.859622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:03.859742 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:03.859945 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:03.859946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:03.859946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:03.860329 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:03.860514 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:03.860584 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:19:03.860821 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260620 14:19:03.863843 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:03.864339 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:03.867990 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81afd470a8364dfdb3f2fb416bd6a389: Generated new TSK 0
I20260620 14:19:03.868173 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:03.894755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:03.981437 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:03.981667 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:03.981706 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:03.982862 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:03.982926 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:03.982972 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:03.984382 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:03.984640 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:03.984856 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:03.989790 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:03.989816 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:03.989948 1025 server_base.cc:1034] running on GCE node
W20260620 14:19:03.989791 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:03.990271 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:03.990468 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:03.991613 1025 hybrid_clock.cc:648] HybridClock initialized: now 1781965143991592 us; error 39 us; skew 500 ppm
Jun 20 14:19:03 dist-test-slave-h6n5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:03.993952 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:03.994362 1025 webserver.cc:460] Webserver started at http://127.0.98.1:38303/ using document root <none> and password file <none>
I20260620 14:19:03.994575 1025 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:03.994640 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:03.994781 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:03.995872 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "84884c237aa34a7392fc514ecb97de3b"
format_stamp: "Formatted at 2026-06-20 14:19:03 on dist-test-slave-h6n5"
server_key: "34a79e60f42055fc25c0104a4537a251"
server_key_iv: "d50fc19a2a45f7fddbd01506e9d4adb0"
server_key_version: "encryptionkey@0"
I20260620 14:19:03.996227 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "84884c237aa34a7392fc514ecb97de3b"
format_stamp: "Formatted at 2026-06-20 14:19:03 on dist-test-slave-h6n5"
server_key: "34a79e60f42055fc25c0104a4537a251"
server_key_iv: "d50fc19a2a45f7fddbd01506e9d4adb0"
server_key_version: "encryptionkey@0"
I20260620 14:19:03.997560 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:03.998333 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:03.998579 1025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:03.998683 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "84884c237aa34a7392fc514ecb97de3b"
format_stamp: "Formatted at 2026-06-20 14:19:03 on dist-test-slave-h6n5"
server_key: "34a79e60f42055fc25c0104a4537a251"
server_key_iv: "d50fc19a2a45f7fddbd01506e9d4adb0"
server_key_version: "encryptionkey@0"
I20260620 14:19:03.998761 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:04.013746 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:04.015570 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:04.015733 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:04.015969 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:04.016330 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:04.016381 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.016422 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:04.016450 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.022400 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46273
I20260620 14:19:04.022442 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46273 every 8 connection(s)
I20260620 14:19:04.022759 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:04.030272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 1025
I20260620 14:19:04.030380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:04.030591 392 external_mini_cluster.cc:1428] Setting key 1e8db44ade0a7fd60fea3a606f1d887b
I20260620 14:19:04.033501 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.023485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36483 (local address 127.0.98.62:39053)
0620 14:19:04.023682 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.023685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.023870 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:19:04.024520 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.024521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.024537 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.024593 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:04.025614 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.026131 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.027595 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.027824 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.027913 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.030191 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:04.030192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:04.030192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:04.030199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:04.031930 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.032449 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.032450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.032451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.032503 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.032840 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.032841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.032841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.032949 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:04.032997 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.033331 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.033433 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260620 14:19:04.033764 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.023472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:36483)
0620 14:19:04.023723 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.023772 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.024345 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.024698 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.024702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.025301 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:04.025499 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.025504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.026227 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.026228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.027375 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.027378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.027922 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.027923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.028060 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.028574 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:04.028583 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:04.030049 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:04.032007 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:04.032015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:04.032017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:04.032352 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:04.032601 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:04.032602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:04.032603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:04.032768 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:04.033008 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:04.033014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:04.033191 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.033314 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.033502 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260620 14:19:04.034307 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39053
I20260620 14:19:04.034396 1154 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:04.034592 1154 heartbeater.cc:511] Master 127.0.98.62:39053 requested a full tablet report, sending...
I20260620 14:19:04.035136 963 ts_manager.cc:194] Registered new tserver with Master: 84884c237aa34a7392fc514ecb97de3b (127.0.98.1:46273)
I20260620 14:19:04.035811 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36483
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:04.056458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:04.136034 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:04.136284 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:04.136324 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:04.137482 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:04.137557 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:04.137603 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:04.139036 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:04.139290 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:04.139509 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:04.144228 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:04.144253 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:04.144515 1159 server_base.cc:1034] running on GCE node
W20260620 14:19:04.144269 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:04.144742 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:04.144960 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:04.146099 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781965144146089 us; error 28 us; skew 500 ppm
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:04.148524 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:04.148977 1159 webserver.cc:460] Webserver started at http://127.0.98.2:39643/ using document root <none> and password file <none>
I20260620 14:19:04.149189 1159 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:04.149255 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:04.149399 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:04.150389 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "e6b64ac16bc14ad7be23bafa8ee99341"
format_stamp: "Formatted at 2026-06-20 14:19:04 on dist-test-slave-h6n5"
server_key: "a8066cde2cc9e99ca7b885e4975ebe8b"
server_key_iv: "2e85c46ea4d64528ee1cf1d75d64d64b"
server_key_version: "encryptionkey@0"
I20260620 14:19:04.150735 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e6b64ac16bc14ad7be23bafa8ee99341"
format_stamp: "Formatted at 2026-06-20 14:19:04 on dist-test-slave-h6n5"
server_key: "a8066cde2cc9e99ca7b885e4975ebe8b"
server_key_iv: "2e85c46ea4d64528ee1cf1d75d64d64b"
server_key_version: "encryptionkey@0"
I20260620 14:19:04.152173 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:19:04.152967 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.153143 1159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:04.153240 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "e6b64ac16bc14ad7be23bafa8ee99341"
format_stamp: "Formatted at 2026-06-20 14:19:04 on dist-test-slave-h6n5"
server_key: "a8066cde2cc9e99ca7b885e4975ebe8b"
server_key_iv: "2e85c46ea4d64528ee1cf1d75d64d64b"
server_key_version: "encryptionkey@0"
I20260620 14:19:04.153321 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:04.173695 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:04.175704 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:04.175868 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:04.176103 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:04.176410 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:04.176466 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.176506 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:04.176535 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.182412 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36441
I20260620 14:19:04.182466 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36441 every 8 connection(s)
I20260620 14:19:04.182761 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:04.191334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 1159
I20260620 14:19:04.191457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:04.191668 392 external_mini_cluster.cc:1428] Setting key 822c46f406e3c3b68d92afcebd7494a1
I20260620 14:19:04.192226 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.183434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37259 (local address 127.0.98.62:39053)
0620 14:19:04.183507 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.183509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.183808 (+ 299us) server_negotiation.cc:407] Connection header received
0620 14:19:04.184262 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.184263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.184273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.184325 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:04.185228 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.185722 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.186797 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.186987 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.187039 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.189203 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:04.189205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:04.189205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:04.189212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:04.190753 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.191130 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.191131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.191132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.191215 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.191502 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.191504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.191504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.191681 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:04.191764 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.192086 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.192162 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:19:04.192514 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.183421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:37259)
0620 14:19:04.183653 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.183701 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.184177 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.184378 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.184381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.184953 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:04.185147 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.185152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.185800 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.185801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.186718 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.186721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.187058 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.187058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.187157 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.187701 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:04.187709 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:04.189101 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:04.190835 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:04.190841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:04.190843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:04.191061 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:04.191295 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:04.191296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:04.191296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:04.191407 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:04.191755 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:04.191761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:04.191948 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.192071 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.192232 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260620 14:19:04.193070 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39053
I20260620 14:19:04.193181 1288 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:04.193377 1288 heartbeater.cc:511] Master 127.0.98.62:39053 requested a full tablet report, sending...
I20260620 14:19:04.193771 963 ts_manager.cc:194] Registered new tserver with Master: e6b64ac16bc14ad7be23bafa8ee99341 (127.0.98.2:36441)
I20260620 14:19:04.194315 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37259
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:04.216996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:04.299414 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:04.299656 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:04.299681 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:04.300884 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:04.300930 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:04.300957 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:04.302481 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:04.302663 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:04.302842 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:04.307701 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:04.307718 1293 server_base.cc:1034] running on GCE node
W20260620 14:19:04.307703 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:04.307720 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:04.308063 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:04.308264 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:04.309432 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781965144309408 us; error 41 us; skew 500 ppm
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:04.311851 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:04.312301 1293 webserver.cc:460] Webserver started at http://127.0.98.3:35389/ using document root <none> and password file <none>
I20260620 14:19:04.312526 1293 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:04.312569 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:04.312719 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:04.313776 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "d437114abdc44750a8a66ab6beaf8fb6"
format_stamp: "Formatted at 2026-06-20 14:19:04 on dist-test-slave-h6n5"
server_key: "cc5039c085618eb69cafd2353560b56e"
server_key_iv: "3cf886421fdeb4e9de76b698d76f7443"
server_key_version: "encryptionkey@0"
I20260620 14:19:04.314138 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d437114abdc44750a8a66ab6beaf8fb6"
format_stamp: "Formatted at 2026-06-20 14:19:04 on dist-test-slave-h6n5"
server_key: "cc5039c085618eb69cafd2353560b56e"
server_key_iv: "3cf886421fdeb4e9de76b698d76f7443"
server_key_version: "encryptionkey@0"
I20260620 14:19:04.315464 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:04.316313 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.316478 1293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:04.316558 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "d437114abdc44750a8a66ab6beaf8fb6"
format_stamp: "Formatted at 2026-06-20 14:19:04 on dist-test-slave-h6n5"
server_key: "cc5039c085618eb69cafd2353560b56e"
server_key_iv: "3cf886421fdeb4e9de76b698d76f7443"
server_key_version: "encryptionkey@0"
I20260620 14:19:04.316632 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:04.337050 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:04.339213 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:04.339334 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:04.339517 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:04.339884 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:04.339920 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.339943 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:04.339998 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:04.345805 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35939
I20260620 14:19:04.345885 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35939 every 8 connection(s)
I20260620 14:19:04.346233 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:04.352010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 1293
I20260620 14:19:04.352098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:04.352296 392 external_mini_cluster.cc:1428] Setting key e67a13eaaf4ba49cb685f81f1f4a9f44
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:04.356436 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.347025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36761 (local address 127.0.98.62:39053)
0620 14:19:04.347112 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.347115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.347301 (+ 186us) server_negotiation.cc:407] Connection header received
0620 14:19:04.347910 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.347912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.347925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.347971 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:04.348927 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.349445 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.350463 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.350679 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.350766 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.353702 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:04.353703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:04.353703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:04.353710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:04.355006 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.355381 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.355382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.355382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.355462 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.355751 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.355751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.355752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.355907 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:04.355980 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.356274 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.356339 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:19:04.356546 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.346988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:36761)
0620 14:19:04.347177 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.347213 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.347791 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.348045 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.348049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.348622 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:04.348819 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.348823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.349545 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.349546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.350360 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.350362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.350777 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.350779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.350911 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.351531 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:04.351543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:04.353507 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:04.355085 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:04.355091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:04.355093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:04.355316 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:04.355521 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:04.355522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:04.355522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:04.355692 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:04.356003 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:04.356008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:04.356152 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.356234 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.356358 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260620 14:19:04.357010 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39053
I20260620 14:19:04.357136 1422 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:04.357321 1422 heartbeater.cc:511] Master 127.0.98.62:39053 requested a full tablet report, sending...
I20260620 14:19:04.357697 963 ts_manager.cc:194] Registered new tserver with Master: d437114abdc44750a8a66ab6beaf8fb6 (127.0.98.3:35939)
I20260620 14:19:04.358086 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36761
I20260620 14:19:04.364194 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:19:04.371801 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.365746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.0.98.62:39053)
0620 14:19:04.365820 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.365823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.365962 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:19:04.365992 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.365993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.366002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.366053 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:04.366646 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.367140 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.367985 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.368220 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.368284 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.369460 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:04.369461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:04.369462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:04.369468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:04.370756 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.371064 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.371065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.371065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.371115 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.371259 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.371260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.371260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.371387 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:04.371461 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.371641 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.371704 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:19:04.374107 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:04.375762 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:04.382565 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.378622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.98.1:46273)
0620 14:19:04.378776 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.378780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.378790 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:04.378832 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.378834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.378850 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.378998 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:04.379180 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.379982 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.381389 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.382145 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.382227 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.382298 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.382436 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.382485 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260620 14:19:04.382553 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.378392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46273 (local address 127.0.0.1:36292)
0620 14:19:04.378593 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.378601 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.378641 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.378951 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.378951 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.378953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:04.379107 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.379108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.380083 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.380084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.381301 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.381303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.382239 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.382240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.382317 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.382334 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.382408 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.382471 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":98,"thread_start_us":76,"threads_started":1}
I20260620 14:19:04.382791 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.378165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35939 (local address 127.0.0.1:59880)
0620 14:19:04.378369 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.378426 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.378496 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.378917 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.378920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.378922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:04.379016 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.379019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.380443 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.380444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.381461 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.381462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.382295 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.382296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.382403 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.382448 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.382484 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.382527 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260620 14:19:04.382815 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.378265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.0.98.3:35939)
0620 14:19:04.378729 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.378732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.378740 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:04.378771 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.378773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.378784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.378911 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:04.379226 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.380339 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.381530 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.382201 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.382602 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.382670 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.382698 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.382741 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":71,"threads_started":1}
I20260620 14:19:04.383507 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 80cefe7bec7045f0b7365d190a30c843 (DEFAULT_TABLE table=test-table [id=e225356021784fa3a9a044a3ba93dae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:04.383512 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 80cefe7bec7045f0b7365d190a30c843 (DEFAULT_TABLE table=test-table [id=e225356021784fa3a9a044a3ba93dae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:04.383915 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.378219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36441 (local address 127.0.0.1:39646)
0620 14:19:04.378532 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.378542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.378829 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.379063 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.379064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.379066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:04.379120 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.379122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.381529 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.381529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.382178 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.382180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.383673 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.383674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.383740 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.383744 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.383797 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.383837 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":79,"threads_started":1}
I20260620 14:19:04.383941 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cefe7bec7045f0b7365d190a30c843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:04.383942 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cefe7bec7045f0b7365d190a30c843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:04.384218 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.378330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.0.98.2:36441)
0620 14:19:04.378548 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.378553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.378844 (+ 291us) server_negotiation.cc:407] Connection header received
0620 14:19:04.378890 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.378891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.378907 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.378978 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:04.379200 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.380336 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.382292 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.383482 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.383557 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.383636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.384101 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.384146 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260620 14:19:04.384814 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 80cefe7bec7045f0b7365d190a30c843 (DEFAULT_TABLE table=test-table [id=e225356021784fa3a9a044a3ba93dae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:04.385124 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80cefe7bec7045f0b7365d190a30c843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:04.387277 1440 tablet_bootstrap.cc:492] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b: Bootstrap starting.
I20260620 14:19:04.387480 1439 tablet_bootstrap.cc:492] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6: Bootstrap starting.
I20260620 14:19:04.387894 1440 tablet_bootstrap.cc:654] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:04.388130 1439 tablet_bootstrap.cc:654] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:04.388291 1440 log.cc:824] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:04.388437 1441 tablet_bootstrap.cc:492] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341: Bootstrap starting.
I20260620 14:19:04.388710 1439 log.cc:824] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:04.388972 1441 tablet_bootstrap.cc:654] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:04.389511 1441 log.cc:824] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:04.389554 1439 tablet_bootstrap.cc:492] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6: No bootstrap required, opened a new log
I20260620 14:19:04.389602 1440 tablet_bootstrap.cc:492] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b: No bootstrap required, opened a new log
I20260620 14:19:04.389616 1439 ts_tablet_manager.cc:1397] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:04.389668 1440 ts_tablet_manager.cc:1397] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:19:04.390278 1441 tablet_bootstrap.cc:492] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341: No bootstrap required, opened a new log
I20260620 14:19:04.390349 1441 ts_tablet_manager.cc:1397] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:04.391474 1440 raft_consensus.cc:348] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.391523 1439 raft_consensus.cc:348] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.391698 1440 raft_consensus.cc:374] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:04.391736 1439 raft_consensus.cc:374] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:04.391743 1440 raft_consensus.cc:729] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84884c237aa34a7392fc514ecb97de3b, State: Initialized, Role: FOLLOWER
I20260620 14:19:04.391772 1439 raft_consensus.cc:729] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d437114abdc44750a8a66ab6beaf8fb6, State: Initialized, Role: FOLLOWER
I20260620 14:19:04.391873 1440 consensus_queue.cc:260] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.391896 1439 consensus_queue.cc:260] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.392165 1439 ts_tablet_manager.cc:1428] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 14:19:04.392185 1422 heartbeater.cc:503] Master 127.0.98.62:39053 was elected leader, sending a full tablet report...
I20260620 14:19:04.392287 1441 raft_consensus.cc:348] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.392445 1441 raft_consensus.cc:374] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:04.392489 1441 raft_consensus.cc:729] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b64ac16bc14ad7be23bafa8ee99341, State: Initialized, Role: FOLLOWER
I20260620 14:19:04.392590 1441 consensus_queue.cc:260] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.392848 1441 ts_tablet_manager.cc:1428] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:19:04.392858 1288 heartbeater.cc:503] Master 127.0.98.62:39053 was elected leader, sending a full tablet report...
I20260620 14:19:04.392513 1154 heartbeater.cc:503] Master 127.0.98.62:39053 was elected leader, sending a full tablet report...
I20260620 14:19:04.392489 1440 ts_tablet_manager.cc:1428] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260620 14:19:04.433316 1289 tablet.cc:2369] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:19:04.523608 1155 tablet.cc:2369] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:04.570834 1446 raft_consensus.cc:482] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:04.571014 1446 raft_consensus.cc:504] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.571385 1446 leader_election.cc:288] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d437114abdc44750a8a66ab6beaf8fb6 (127.0.98.3:35939), e6b64ac16bc14ad7be23bafa8ee99341 (127.0.98.2:36441)
I20260620 14:19:04.574489 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.571683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35939 (local address 127.0.98.1:50541)
0620 14:19:04.571826 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.571835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.571922 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.572049 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.572049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.572051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:04.572153 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.572156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.572821 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.572822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.573658 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.573660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.574213 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.574214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.574287 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.574293 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.574347 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.574384 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":26,"thread_start_us":60,"threads_started":1}
I20260620 14:19:04.574524 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.571708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50541 (local address 127.0.98.3:35939)
0620 14:19:04.571774 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.571776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.571935 (+ 159us) server_negotiation.cc:407] Connection header received
0620 14:19:04.571972 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.571974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.571988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.572045 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:04.572238 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.572734 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.573755 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.574130 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.574184 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.574193 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.574368 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.574408 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:19:04.574637 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.571675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40373 (local address 127.0.98.2:36441)
0620 14:19:04.571723 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.571725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.571882 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:19:04.571920 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.571921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.571936 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.571994 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:04.572185 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.572694 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.573758 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.574238 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.574309 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.574318 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.574493 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.574542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:19:04.574616 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.571643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36441 (local address 127.0.98.1:40373)
0620 14:19:04.571782 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:04.571795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:04.571860 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:04.572003 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:04.572005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:04.572026 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:04.572106 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.572108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.572800 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.572801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.573657 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:04.573659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.574316 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.574317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:04.574406 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.574410 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:04.574469 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:19:04.574505 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":28,"thread_start_us":76,"threads_started":1}
I20260620 14:19:04.575095 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cefe7bec7045f0b7365d190a30c843" candidate_uuid: "84884c237aa34a7392fc514ecb97de3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" is_pre_election: true
I20260620 14:19:04.575258 1242 raft_consensus.cc:2454] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84884c237aa34a7392fc514ecb97de3b in term 0.
I20260620 14:19:04.575249 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cefe7bec7045f0b7365d190a30c843" candidate_uuid: "84884c237aa34a7392fc514ecb97de3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d437114abdc44750a8a66ab6beaf8fb6" is_pre_election: true
I20260620 14:19:04.575426 1376 raft_consensus.cc:2454] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84884c237aa34a7392fc514ecb97de3b in term 0.
I20260620 14:19:04.575536 1044 leader_election.cc:302] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84884c237aa34a7392fc514ecb97de3b, e6b64ac16bc14ad7be23bafa8ee99341; no voters:
I20260620 14:19:04.575685 1446 raft_consensus.cc:2781] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:04.575752 1446 raft_consensus.cc:482] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:04.575778 1446 raft_consensus.cc:3037] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:04.576519 1446 raft_consensus.cc:504] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.576658 1446 leader_election.cc:288] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [CANDIDATE]: Term 1 election: Requested vote from peers d437114abdc44750a8a66ab6beaf8fb6 (127.0.98.3:35939), e6b64ac16bc14ad7be23bafa8ee99341 (127.0.98.2:36441)
I20260620 14:19:04.576807 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cefe7bec7045f0b7365d190a30c843" candidate_uuid: "84884c237aa34a7392fc514ecb97de3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d437114abdc44750a8a66ab6beaf8fb6"
I20260620 14:19:04.576819 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80cefe7bec7045f0b7365d190a30c843" candidate_uuid: "84884c237aa34a7392fc514ecb97de3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b64ac16bc14ad7be23bafa8ee99341"
I20260620 14:19:04.576902 1376 raft_consensus.cc:3037] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:04.576907 1242 raft_consensus.cc:3037] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:04.577605 1376 raft_consensus.cc:2454] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84884c237aa34a7392fc514ecb97de3b in term 1.
I20260620 14:19:04.577656 1242 raft_consensus.cc:2454] T 80cefe7bec7045f0b7365d190a30c843 P e6b64ac16bc14ad7be23bafa8ee99341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84884c237aa34a7392fc514ecb97de3b in term 1.
I20260620 14:19:04.577827 1044 leader_election.cc:302] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84884c237aa34a7392fc514ecb97de3b, e6b64ac16bc14ad7be23bafa8ee99341; no voters:
I20260620 14:19:04.577925 1446 raft_consensus.cc:2781] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:04.578097 1446 raft_consensus.cc:686] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [term 1 LEADER]: Becoming Leader. State: Replica: 84884c237aa34a7392fc514ecb97de3b, State: Running, Role: LEADER
I20260620 14:19:04.578194 1446 consensus_queue.cc:237] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } }
I20260620 14:19:04.579020 963 catalog_manager.cc:5495] T 80cefe7bec7045f0b7365d190a30c843 P 84884c237aa34a7392fc514ecb97de3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84884c237aa34a7392fc514ecb97de3b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "84884c237aa34a7392fc514ecb97de3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437114abdc44750a8a66ab6beaf8fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84884c237aa34a7392fc514ecb97de3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6b64ac16bc14ad7be23bafa8ee99341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36441 } health_report { overall_health: UNKNOWN } } }
W20260620 14:19:04.596777 1423 tablet.cc:2369] T 80cefe7bec7045f0b7365d190a30c843 P d437114abdc44750a8a66ab6beaf8fb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:04.615072 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.607322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.0.98.1:46273)
0620 14:19:04.607413 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.607415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.607527 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:19:04.607641 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.607642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.607658 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.607728 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:04.608293 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.608771 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.609492 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.609692 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.609771 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.612612 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:04.612623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:04.612624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:04.612631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:04.614094 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.614409 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.614412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.614412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.614465 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.614652 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.614653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.614653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.614786 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:04.614897 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.614947 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.615005 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:04.637632 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.630783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.0.98.1:46273)
0620 14:19:04.630858 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.630860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.630975 (+ 115us) server_negotiation.cc:407] Connection header received
0620 14:19:04.631093 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.631095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.631106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.631185 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:04.631821 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.632295 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.633019 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.633215 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.633300 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.635447 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:04.635448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:04.635449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:04.635456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:04.636704 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.636998 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.636998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.636999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.637046 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.637200 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.637200 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.637201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.637285 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:04.637358 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.637489 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.637568 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:04.659960 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:04.652806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.0.98.1:46273)
0620 14:19:04.652881 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:04.652883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:04.652952 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:19:04.652989 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:04.652990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:04.653001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:04.653074 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:04.653635 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.654118 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.654800 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:04.654974 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:04.655026 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:04.657338 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:04.657339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:04.657339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:04.657346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:04.658822 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.659157 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.659158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.659158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.659209 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:04.659420 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:04.659421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:04.659421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:04.659509 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:04.659659 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:04.659794 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:19:04.659851 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:19:04.660455 1068 tablet_service.cc:2908] Rejecting scan request for tablet 80cefe7bec7045f0b7365d190a30c843: Uninitialized: clean time has not yet been initialized
I20260620 14:19:04.661301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 1025
I20260620 14:19:04.666808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 1159
I20260620 14:19:04.672153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 1293
I20260620 14:19:04.676826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 931
2026-06-20T14:19:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2124 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[1464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[1464](info): set up 2 sockets
Jun 20 14:19:04 dist-test-slave-h6n5 krb5kdc[1464](info): commencing operation
krb5kdc: starting...
W20260620 14:19:05.715160 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:05 dist-test-slave-h6n5 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260620 14:19:06.411128 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.597s user 0.000s sys 0.003s
I20260620 14:19:06.411511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:06.493484 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:06.493722 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:06.493745 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:06.494959 1477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:06.495005 1477 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:19:06.495019 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:06.495033 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:06.495044 1477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:06.495056 1477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:06.496639 1477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38145
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:06.496897 1477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:06.497136 1477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:06.502229 1482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:06.502202 1483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:06.502198 1485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:06.502349 1477 server_base.cc:1034] running on GCE node
I20260620 14:19:06.502681 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:06.502921 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:06.504055 1477 hybrid_clock.cc:648] HybridClock initialized: now 1781965146504045 us; error 43 us; skew 500 ppm
Jun 20 14:19:06 dist-test-slave-h6n5 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:06.509630 1477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:06.510072 1477 webserver.cc:460] Webserver started at http://127.0.98.62:35677/ using document root <none> and password file <none>
I20260620 14:19:06.510257 1477 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:06.510296 1477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:06.510385 1477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:06.511399 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "aae9b47c9a354979b93ec894b88c6e5d"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "e98226555e717d0b6a23a5626292d4fe"
server_key_iv: "169401a1b0b541cb2e8ace06d6c6f5aa"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.511759 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "aae9b47c9a354979b93ec894b88c6e5d"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "e98226555e717d0b6a23a5626292d4fe"
server_key_iv: "169401a1b0b541cb2e8ace06d6c6f5aa"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.513003 1477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:06.513855 1492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:06.514096 1477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:06.514213 1477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "aae9b47c9a354979b93ec894b88c6e5d"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "e98226555e717d0b6a23a5626292d4fe"
server_key_iv: "169401a1b0b541cb2e8ace06d6c6f5aa"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.514295 1477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:06.525949 1477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:06.528213 1477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:06.528321 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:06.532845 1477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38145
I20260620 14:19:06.532887 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38145 every 8 connection(s)
I20260620 14:19:06.533206 1477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:06.533860 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:06.536202 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d: Bootstrap starting.
I20260620 14:19:06.536377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 1477
I20260620 14:19:06.536468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:06.536667 392 external_mini_cluster.cc:1428] Setting key c3a80c7f745b572140098f4848b8fed4
I20260620 14:19:06.536916 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:06.537272 1555 log.cc:824] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:06.537971 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d: No bootstrap required, opened a new log
I20260620 14:19:06.539300 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae9b47c9a354979b93ec894b88c6e5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:19:06.539460 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:06.539506 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae9b47c9a354979b93ec894b88c6e5d, State: Initialized, Role: FOLLOWER
I20260620 14:19:06.539662 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae9b47c9a354979b93ec894b88c6e5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:19:06.539762 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:06.539806 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:06.539855 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:06.540364 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae9b47c9a354979b93ec894b88c6e5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:19:06.540486 1555 leader_election.cc:302] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae9b47c9a354979b93ec894b88c6e5d; no voters:
I20260620 14:19:06.540680 1555 leader_election.cc:288] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:06.540702 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:06.540828 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [term 1 LEADER]: Becoming Leader. State: Replica: aae9b47c9a354979b93ec894b88c6e5d, State: Running, Role: LEADER
I20260620 14:19:06.540940 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:06.540983 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae9b47c9a354979b93ec894b88c6e5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260620 14:19:06.541369 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae9b47c9a354979b93ec894b88c6e5d. Latest consensus state: current_term: 1 leader_uuid: "aae9b47c9a354979b93ec894b88c6e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae9b47c9a354979b93ec894b88c6e5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } } }
I20260620 14:19:06.541378 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae9b47c9a354979b93ec894b88c6e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae9b47c9a354979b93ec894b88c6e5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } } }
I20260620 14:19:06.541553 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:06.541565 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:06.541874 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:06.542431 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:19:06 dist-test-slave-h6n5 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:06.545706 1569 catalog_manager.cc:1347] Generated new cluster ID: c806b8ab0b0a4444a8735f200897c31b
I20260620 14:19:06.545814 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:06.559880 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.537137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.0.98.62:38145)
0620 14:19:06.537296 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:06.537306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:06.537338 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:19:06.537801 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:06.537808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:06.537854 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:06.538019 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:06.538630 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.539969 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.540879 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.541445 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.541546 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.545701 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:06.545705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:06.545705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:06.545717 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:06.557652 (+ 11935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.558141 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.558145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.558145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.558261 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.558524 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.558525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.558525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.558847 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:06.559004 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:06.559449 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 14:19:06.559627 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260620 14:19:06.561107 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:06.561656 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:06.570470 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d: Generated new TSK 0
I20260620 14:19:06.570691 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:06.604561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:06.686091 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:06.686300 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:06.686326 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:06.687534 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:06.687606 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:06.687628 1581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:06.687644 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:06.689211 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:06.689455 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:06.689707 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:06.695122 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:06.695151 1581 server_base.cc:1034] running on GCE node
W20260620 14:19:06.695138 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:06.695130 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:06.695514 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:06.695793 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:06.696938 1581 hybrid_clock.cc:648] HybridClock initialized: now 1781965146696911 us; error 43 us; skew 500 ppm
Jun 20 14:19:06 dist-test-slave-h6n5 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:06.699656 1581 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:06.700075 1581 webserver.cc:460] Webserver started at http://127.0.98.1:41583/ using document root <none> and password file <none>
I20260620 14:19:06.700274 1581 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:06.700315 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:06.700404 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:06.701436 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "1622ef25ba9e44249c4f5843d6a3ec72"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "5e7fc01c246293bcd6ce4aa0b0c499e3"
server_key_iv: "1de56d535fc4427cc6373c2e73820819"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.701763 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1622ef25ba9e44249c4f5843d6a3ec72"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "5e7fc01c246293bcd6ce4aa0b0c499e3"
server_key_iv: "1de56d535fc4427cc6373c2e73820819"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.703028 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:06.703751 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:06.703958 1581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:06.704061 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "1622ef25ba9e44249c4f5843d6a3ec72"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "5e7fc01c246293bcd6ce4aa0b0c499e3"
server_key_iv: "1de56d535fc4427cc6373c2e73820819"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.704149 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:06.713784 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:06.715867 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:06.716030 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:06.716533 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:06.716598 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:06.716645 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:06.716670 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:19:06 dist-test-slave-h6n5 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:06.723888 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37629
I20260620 14:19:06.723979 1711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37629 every 8 connection(s)
I20260620 14:19:06.724354 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:06.726891 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.717062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37445 (local address 127.0.98.62:38145)
0620 14:19:06.717103 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:06.717105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:06.717360 (+ 255us) server_negotiation.cc:407] Connection header received
0620 14:19:06.717897 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:06.717898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:06.717929 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:06.717977 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:06.718966 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.719865 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.720984 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.721344 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.721428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.723811 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:06.723813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:06.723814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:06.723823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:06.725414 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.725800 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.725802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.725802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.725857 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.726122 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.726122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.726123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.726228 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:06.726281 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:06.726743 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 14:19:06.726824 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:19:06.727106 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.717081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.1:37445)
0620 14:19:06.717240 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:06.717274 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:06.717792 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:06.718072 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:06.718076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:06.718496 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:06.718815 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.718823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.719964 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.719965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.720892 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.720894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.721442 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.721443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.721549 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.722119 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:06.722127 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:06.723709 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:06.725484 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.725489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.725490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.725719 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.725943 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.725944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.725944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.726052 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.726322 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:06.726334 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:06.726454 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:19:06.726680 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 14:19:06.726836 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260620 14:19:06.727723 1712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38145
I20260620 14:19:06.727816 1712 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:06.728021 1712 heartbeater.cc:511] Master 127.0.98.62:38145 requested a full tablet report, sending...
I20260620 14:19:06.728826 1508 ts_manager.cc:194] Registered new tserver with Master: 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629)
I20260620 14:19:06.729804 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37445
I20260620 14:19:06.730199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 1581
I20260620 14:19:06.730306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:06.730504 392 external_mini_cluster.cc:1428] Setting key 7455ea360e48b996fce4608a9aeeb3c9
I20260620 14:19:06.735219 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.729022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.1:33593)
0620 14:19:06.729158 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:06.729170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:06.729236 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:06.729350 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:06.729351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:06.729555 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:06.729615 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.729616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.730375 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.730376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.731180 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.731183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.731577 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.731578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.731639 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.732127 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:06.732131 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:06.732415 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:06.734275 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.734276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.734276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.734442 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.734640 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.734641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.734641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.734684 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.734961 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:06.734961 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:06.735012 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:19:06.735069 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:19:06.735151 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260620 14:19:06.735414 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.729104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33593 (local address 127.0.98.62:38145)
0620 14:19:06.729146 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:06.729148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:06.729218 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:19:06.729291 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:06.729292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:06.729299 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:06.729344 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:06.729697 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.730250 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.731257 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.731464 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.731539 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.732540 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:06.732543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:06.732544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:06.732555 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:06.734199 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.734514 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.734515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.734516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.734577 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.734739 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.734740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.734740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.734888 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:06.734958 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:06.735123 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:19:06.735206 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:06.757987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:06.839845 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:06.840065 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:06.840091 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:06.841297 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:06.841341 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:06.841362 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:06.841380 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:06.843115 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:06.843309 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:06.843482 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:06.848352 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:06.848377 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:06.848421 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:06.848583 1721 server_base.cc:1034] running on GCE node
I20260620 14:19:06.848820 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:06.849064 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:06.850229 1721 hybrid_clock.cc:648] HybridClock initialized: now 1781965146850213 us; error 47 us; skew 500 ppm
Jun 20 14:19:06 dist-test-slave-h6n5 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:06.852753 1721 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:06.853225 1721 webserver.cc:460] Webserver started at http://127.0.98.2:41713/ using document root <none> and password file <none>
I20260620 14:19:06.853451 1721 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:06.853528 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:06.853678 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:06.854733 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8812f059b3e476ab29e64dc0177d1d2"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "79e2154b0da1ec5aa09a1fc1c77bee27"
server_key_iv: "52c298af527472a12692fdaba313edff"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.855094 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8812f059b3e476ab29e64dc0177d1d2"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "79e2154b0da1ec5aa09a1fc1c77bee27"
server_key_iv: "52c298af527472a12692fdaba313edff"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.856503 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:06.857290 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:06.857461 1721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:06.857587 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "c8812f059b3e476ab29e64dc0177d1d2"
format_stamp: "Formatted at 2026-06-20 14:19:06 on dist-test-slave-h6n5"
server_key: "79e2154b0da1ec5aa09a1fc1c77bee27"
server_key_iv: "52c298af527472a12692fdaba313edff"
server_key_version: "encryptionkey@0"
I20260620 14:19:06.857663 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:06.866822 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:06.868906 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:06.869071 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:06.869657 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:06.869709 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:06.869740 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:06.869761 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:19:06 dist-test-slave-h6n5 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:06.876986 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34205
I20260620 14:19:06.877115 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34205 every 8 connection(s)
I20260620 14:19:06.877363 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:06.879323 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.870053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42737 (local address 127.0.98.62:38145)
0620 14:19:06.870118 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:06.870121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:06.870316 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:19:06.870886 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:06.870887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:06.870901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:06.870982 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:06.871686 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.872186 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.873252 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.873490 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.873579 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.876146 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:06.876147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:06.876148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:06.876157 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:06.877908 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.878317 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.878319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.878319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.878376 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.878662 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.878663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.878663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.878780 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:06.878855 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:06.879134 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:19:06.879227 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:19:06.879503 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.870039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.2:42737)
0620 14:19:06.870200 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:06.870235 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:06.870778 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:06.871060 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:06.871063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:06.871354 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:06.871541 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.871588 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.872288 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.872289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.873166 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.873169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.873587 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.873588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.873714 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.874241 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:06.874249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:06.876067 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:06.878003 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.878009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.878011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.878233 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.878479 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.878480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.878480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.878584 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.878846 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:06.878852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:06.878985 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:19:06.879112 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:19:06.879283 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260620 14:19:06.879984 1852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38145
I20260620 14:19:06.880065 1852 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:06.880239 1852 heartbeater.cc:511] Master 127.0.98.62:38145 requested a full tablet report, sending...
I20260620 14:19:06.880654 1508 ts_manager.cc:194] Registered new tserver with Master: c8812f059b3e476ab29e64dc0177d1d2 (127.0.98.2:34205)
I20260620 14:19:06.881136 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42737
I20260620 14:19:06.883165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 1721
I20260620 14:19:06.883285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:06.883446 392 external_mini_cluster.cc:1428] Setting key 53c83f61278bc6708ab035ebed51c40d
I20260620 14:19:06.887063 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.881038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.2:58847)
0620 14:19:06.881158 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:06.881169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:06.881231 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:06.881356 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:06.881356 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:06.881560 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:06.881614 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.881616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.882362 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.882363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.883044 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:06.883046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.883581 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.883582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:06.883668 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.884107 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:06.884112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:06.884414 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:06.885962 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.885963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.885963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.886199 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.886380 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:06.886381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:06.886381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:06.886447 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:06.886835 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:06.886835 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:06.886869 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:19:06.886919 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:19:06.886993 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260620 14:19:06.887244 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:06.881109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58847 (local address 127.0.98.62:38145)
0620 14:19:06.881151 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:06.881153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:06.881235 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:19:06.881285 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:06.881286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:06.881297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:06.881358 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:06.881687 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.882235 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.883188 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:06.883461 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:06.883545 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:06.884498 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:06.884500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:06.884500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:06.884507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:06.885878 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.886272 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.886272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.886273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.886324 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:06.886531 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:06.886532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:06.886532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:06.886763 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:06.886841 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:06.886959 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:19:06.887040 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:06.907084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38145
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:06.989660 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:06.989884 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:06.989923 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:06.991066 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:06.991129 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:06.991164 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:06.991204 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:06.992663 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:06.992928 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:06.993158 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:06.998112 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:06.998127 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:06.998154 1861 server_base.cc:1034] running on GCE node
W20260620 14:19:06.998119 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:06.998447 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:06.998728 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:06.999868 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781965146999855 us; error 29 us; skew 500 ppm
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:07.006855 1861 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:07.007290 1861 webserver.cc:460] Webserver started at http://127.0.98.3:37251/ using document root <none> and password file <none>
I20260620 14:19:07.007601 1861 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:07.007658 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:07.007810 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:07.008879 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e6262240e13482aa9cbe916167aebf6"
format_stamp: "Formatted at 2026-06-20 14:19:07 on dist-test-slave-h6n5"
server_key: "aa66141cfae89475f39d2e2b7d5e4a44"
server_key_iv: "19c988a774008cb415182c2a6b018f45"
server_key_version: "encryptionkey@0"
I20260620 14:19:07.009239 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e6262240e13482aa9cbe916167aebf6"
format_stamp: "Formatted at 2026-06-20 14:19:07 on dist-test-slave-h6n5"
server_key: "aa66141cfae89475f39d2e2b7d5e4a44"
server_key_iv: "19c988a774008cb415182c2a6b018f45"
server_key_version: "encryptionkey@0"
I20260620 14:19:07.010529 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:07.011257 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.011406 1861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:07.011494 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "1e6262240e13482aa9cbe916167aebf6"
format_stamp: "Formatted at 2026-06-20 14:19:07 on dist-test-slave-h6n5"
server_key: "aa66141cfae89475f39d2e2b7d5e4a44"
server_key_iv: "19c988a774008cb415182c2a6b018f45"
server_key_version: "encryptionkey@0"
I20260620 14:19:07.011540 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:07.026607 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:07.028610 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:07.028715 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:07.029138 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:07.029202 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.029244 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:07.029268 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.035713 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39261
I20260620 14:19:07.035797 1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39261 every 8 connection(s)
I20260620 14:19:07.036108 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:07.039597 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.029617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55971 (local address 127.0.98.62:38145)
0620 14:19:07.029679 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.029682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.029906 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:19:07.030366 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.030368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.030381 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.030452 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:07.031297 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.032156 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.033730 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.034049 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.034152 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.036832 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:07.036834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:07.036834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:07.036860 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:07.038414 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.038803 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.038804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.038804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.038851 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.039055 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.039056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.039056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.039162 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:07.039211 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.039435 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.039490 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:19:07.039844 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.029614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.3:55971)
0620 14:19:07.029774 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.029811 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.030271 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.030547 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.030550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.030923 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:07.031187 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.031193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.032255 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.032256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.033636 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.033639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.034150 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.034151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.034244 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.034896 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:07.034907 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:07.036750 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:07.038496 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:07.038501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:07.038503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:07.038739 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:07.038909 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:07.038910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:07.038910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:07.039003 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:07.039218 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:07.039222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:07.039332 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.039415 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.039531 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260620 14:19:07.040314 1992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38145
I20260620 14:19:07.040410 1992 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:07.040591 1992 heartbeater.cc:511] Master 127.0.98.62:38145 requested a full tablet report, sending...
I20260620 14:19:07.040983 1509 ts_manager.cc:194] Registered new tserver with Master: 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3:39261)
I20260620 14:19:07.041567 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55971
I20260620 14:19:07.041884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 1861
I20260620 14:19:07.042026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:07.042228 392 external_mini_cluster.cc:1428] Setting key 804c3e36d0c2be5fd9b704015774606e
I20260620 14:19:07.043074 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:19:07.047852 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.041308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50479 (local address 127.0.98.62:38145)
0620 14:19:07.041347 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.041349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.041472 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:19:07.041532 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.041533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.041540 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.041583 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:07.041988 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.042827 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.043964 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.044141 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.044195 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.045194 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:07.045196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:07.045196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:07.045203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:07.046785 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.047110 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.047111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.047112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.047175 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.047363 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.047365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.047365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.047470 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:07.047547 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.047704 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.047769 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:19:07.047864 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.041272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.98.3:50479)
0620 14:19:07.041407 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.041418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.041478 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.041591 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.041592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.041813 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:07.041881 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.041883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.042970 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.042970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.043887 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.043890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.044219 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.044220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.044295 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.044699 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:07.044705 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:07.045119 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:07.046870 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:07.046871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:07.046872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:07.047030 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:07.047243 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:07.047243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:07.047244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:07.047291 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:07.047539 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:07.047540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:07.047595 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.047670 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.047763 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:07.076865 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.068744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.0.98.62:38145)
0620 14:19:07.068791 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.068794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.068803 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:07.068832 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.068833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.068843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.068928 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:07.069836 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.070705 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.071685 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.071910 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.072005 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.074446 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:07.074448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:07.074448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:07.074455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:07.075899 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.076267 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.076267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.076268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.076315 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.076484 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.076484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.076485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.076576 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:07.076646 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.076748 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.076801 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:19:07.078598 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:07.080442 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:07.086701 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.082819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37629 (local address 127.0.0.1:57230)
0620 14:19:07.082973 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.083013 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.083077 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.083346 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.083349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.083351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:07.083411 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.083412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.084408 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.084409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.085340 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.085342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.086386 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.086387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.086477 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.086493 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.086577 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.086629 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":58,"threads_started":1}
I20260620 14:19:07.086745 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.082922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.0.98.1:37629)
0620 14:19:07.083124 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.083130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.083139 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:07.083179 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.083181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.083198 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.083271 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:07.083541 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.084321 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.085544 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.086287 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.086369 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.086453 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.086608 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.086649 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260620 14:19:07.087432 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.082833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39261 (local address 127.0.0.1:60926)
0620 14:19:07.083131 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.083144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.083197 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.083642 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.083643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.083645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:07.083693 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.083694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.085175 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.085176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.086319 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.086322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.087251 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.087252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.087311 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.087314 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.087346 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.087387 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260620 14:19:07.087877 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.082946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.0.98.3:39261)
0620 14:19:07.083423 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.083427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.083438 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:07.083480 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.083482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.083499 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.083772 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:07.083832 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.085038 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.086453 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.087130 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.087597 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.087681 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.087710 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.087767 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":74,"threads_started":1}
I20260620 14:19:07.087862 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 78b626f5aa9043dd9e92718778223721 (DEFAULT_TABLE table=test-table [id=b11166c662cb4b10abe5ba430ff20592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:07.088230 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b626f5aa9043dd9e92718778223721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.088454 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 78b626f5aa9043dd9e92718778223721 (DEFAULT_TABLE table=test-table [id=b11166c662cb4b10abe5ba430ff20592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:07.088774 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b626f5aa9043dd9e92718778223721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.091964 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.083017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34205 (local address 127.0.0.1:59036)
0620 14:19:07.083232 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.083242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.083283 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.083699 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.083700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.083702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:07.083739 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.083740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.085136 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.085137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.086325 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.086327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.087310 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.087311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.091742 (+ 4431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.091747 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.091843 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.091888 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":11,"thread_start_us":72,"threads_started":1}
I20260620 14:19:07.092026 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.083240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.0.98.2:34205)
0620 14:19:07.083422 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.083427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.083438 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:07.083480 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.083482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.083499 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.083632 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:07.083820 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.085043 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.086412 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.087216 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.087309 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.087385 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.091867 (+ 4482us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.091916 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260620 14:19:07.092298 2015 tablet_bootstrap.cc:492] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Bootstrap starting.
I20260620 14:19:07.092698 2016 tablet_bootstrap.cc:492] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Bootstrap starting.
I20260620 14:19:07.093065 2015 tablet_bootstrap.cc:654] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.093137 2016 tablet_bootstrap.cc:654] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.093077 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 78b626f5aa9043dd9e92718778223721 (DEFAULT_TABLE table=test-table [id=b11166c662cb4b10abe5ba430ff20592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:07.093350 2016 log.cc:824] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:07.093392 2015 log.cc:824] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:07.093423 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b626f5aa9043dd9e92718778223721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.094115 2015 tablet_bootstrap.cc:492] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: No bootstrap required, opened a new log
I20260620 14:19:07.094121 2016 tablet_bootstrap.cc:492] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: No bootstrap required, opened a new log
I20260620 14:19:07.094175 2015 ts_tablet_manager.cc:1397] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:07.094185 2016 ts_tablet_manager.cc:1397] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:07.095498 2015 raft_consensus.cc:348] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.095693 2015 raft_consensus.cc:374] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.095721 2015 raft_consensus.cc:729] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6262240e13482aa9cbe916167aebf6, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.095830 2015 consensus_queue.cc:260] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.096104 1992 heartbeater.cc:503] Master 127.0.98.62:38145 was elected leader, sending a full tablet report...
I20260620 14:19:07.096117 2015 ts_tablet_manager.cc:1428] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:07.096485 2016 raft_consensus.cc:348] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.096653 2016 raft_consensus.cc:374] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.096689 2016 raft_consensus.cc:729] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1622ef25ba9e44249c4f5843d6a3ec72, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.096799 2016 consensus_queue.cc:260] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.097060 2016 ts_tablet_manager.cc:1428] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:19:07.097112 1712 heartbeater.cc:503] Master 127.0.98.62:38145 was elected leader, sending a full tablet report...
I20260620 14:19:07.097561 2021 tablet_bootstrap.cc:492] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Bootstrap starting.
I20260620 14:19:07.098176 2021 tablet_bootstrap.cc:654] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.098543 2021 log.cc:824] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:07.099298 2021 tablet_bootstrap.cc:492] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: No bootstrap required, opened a new log
I20260620 14:19:07.099352 2021 ts_tablet_manager.cc:1397] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:07.100809 2021 raft_consensus.cc:348] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.100925 2021 raft_consensus.cc:374] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.100946 2021 raft_consensus.cc:729] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8812f059b3e476ab29e64dc0177d1d2, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.101056 2021 consensus_queue.cc:260] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.101270 2021 ts_tablet_manager.cc:1428] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:19:07.101374 1852 heartbeater.cc:503] Master 127.0.98.62:38145 was elected leader, sending a full tablet report...
W20260620 14:19:07.128064 1853 tablet.cc:2369] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:07.188596 2023 raft_consensus.cc:482] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:07.188724 2023 raft_consensus.cc:504] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.189170 2023 leader_election.cc:288] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629), 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3:39261)
I20260620 14:19:07.192221 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.189410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38355 (local address 127.0.98.1:37629)
0620 14:19:07.189466 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.189468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.189838 (+ 370us) server_negotiation.cc:407] Connection header received
0620 14:19:07.189913 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.189914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.189925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.189965 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:07.190131 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.190571 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.191472 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.191847 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.191930 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.191941 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.192114 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.192147 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:19:07.192230 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.189461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48055 (local address 127.0.98.3:39261)
0620 14:19:07.189532 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.189535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.189552 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:19:07.189585 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.189586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.189595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.189644 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:07.189845 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.190497 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.191452 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.191883 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.191969 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.191979 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.192128 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.192164 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:19:07.192251 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.189354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39261 (local address 127.0.98.2:48055)
0620 14:19:07.189414 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.189427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.189475 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.189683 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.189684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.189686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:07.189757 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.189758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.190584 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.190585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.191329 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.191330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.191969 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.191969 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.192046 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.192048 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.192122 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.192161 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:19:07.192255 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.189356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37629 (local address 127.0.98.2:38355)
0620 14:19:07.189762 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.189771 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.189846 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.189989 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.189989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.189991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:07.190060 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.190061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.190667 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.190668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.191378 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.191380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.191943 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.191944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.192023 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.192026 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.192093 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.192134 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":175,"thread_start_us":148,"threads_started":1}
I20260620 14:19:07.192631 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b626f5aa9043dd9e92718778223721" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" is_pre_election: true
I20260620 14:19:07.192636 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b626f5aa9043dd9e92718778223721" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6262240e13482aa9cbe916167aebf6" is_pre_election: true
I20260620 14:19:07.192788 1946 raft_consensus.cc:2454] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 0.
I20260620 14:19:07.192788 1666 raft_consensus.cc:2454] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 0.
I20260620 14:19:07.193007 1739 leader_election.cc:302] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6262240e13482aa9cbe916167aebf6, c8812f059b3e476ab29e64dc0177d1d2; no voters:
I20260620 14:19:07.193158 2023 raft_consensus.cc:2781] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:07.193213 2023 raft_consensus.cc:482] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:07.193240 2023 raft_consensus.cc:3037] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.194129 2023 raft_consensus.cc:504] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.194267 2023 leader_election.cc:288] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 election: Requested vote from peers 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629), 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3:39261)
I20260620 14:19:07.194411 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b626f5aa9043dd9e92718778223721" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622ef25ba9e44249c4f5843d6a3ec72"
I20260620 14:19:07.194486 1666 raft_consensus.cc:3037] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.194470 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b626f5aa9043dd9e92718778223721" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6262240e13482aa9cbe916167aebf6"
I20260620 14:19:07.194591 1946 raft_consensus.cc:3037] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.195226 1666 raft_consensus.cc:2454] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 1.
I20260620 14:19:07.195294 1946 raft_consensus.cc:2454] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 1.
I20260620 14:19:07.195428 1737 leader_election.cc:302] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1622ef25ba9e44249c4f5843d6a3ec72, c8812f059b3e476ab29e64dc0177d1d2; no voters:
I20260620 14:19:07.195580 2023 raft_consensus.cc:2781] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:07.195745 2023 raft_consensus.cc:686] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 LEADER]: Becoming Leader. State: Replica: c8812f059b3e476ab29e64dc0177d1d2, State: Running, Role: LEADER
I20260620 14:19:07.195847 2023 consensus_queue.cc:237] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } }
I20260620 14:19:07.196693 1509 catalog_manager.cc:5495] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8812f059b3e476ab29e64dc0177d1d2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } health_report { overall_health: HEALTHY } } }
I20260620 14:19:07.202907 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.200423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.0.98.2:34205)
0620 14:19:07.200470 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.200472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.200554 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:19:07.200672 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.200673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.200697 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.200764 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:07.200872 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.201313 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.202107 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.202295 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.202367 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.202576 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.202669 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.202769 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.202810 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.202831 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:07.204527 1666 raft_consensus.cc:1261] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Refusing update from remote peer c8812f059b3e476ab29e64dc0177d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:07.204597 1946 raft_consensus.cc:1261] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Refusing update from remote peer c8812f059b3e476ab29e64dc0177d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:07.204865 2023 consensus_queue.cc:1035] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:07.205024 2025 consensus_queue.cc:1035] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:07.207928 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298929242947334144 to 7298929242914803712. Current Snapshot: MvccSnapshot[applied={T|T < 7298929242947334144 or (T in {7298929242947334144})}]
I20260620 14:19:07.210786 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:07.210872 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:07.211620 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d: Sending DeleteTablet for 3 replicas of tablet 78b626f5aa9043dd9e92718778223721
I20260620 14:19:07.211913 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b626f5aa9043dd9e92718778223721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57230
I20260620 14:19:07.211954 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b626f5aa9043dd9e92718778223721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60926
I20260620 14:19:07.212006 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b626f5aa9043dd9e92718778223721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59036
I20260620 14:19:07.212193 2040 tablet_replica.cc:330] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: stopping tablet replica
I20260620 14:19:07.212289 2040 raft_consensus.cc:2229] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:07.212404 2040 raft_consensus.cc:2258] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:07.212390 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:07.212517 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:07.212960 2038 tablet_replica.cc:330] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: stopping tablet replica
I20260620 14:19:07.213018 2040 ts_tablet_manager.cc:1905] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:07.213052 2038 raft_consensus.cc:2229] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:07.213151 2038 raft_consensus.cc:2258] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:07.213296 2039 tablet_replica.cc:330] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: stopping tablet replica
I20260620 14:19:07.213374 2039 raft_consensus.cc:2229] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:07.213460 2039 raft_consensus.cc:2258] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:07.213601 2038 ts_tablet_manager.cc:1905] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:07.213924 2039 ts_tablet_manager.cc:1905] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:07.214270 1786 tablet_service.cc:1467] Processing CreateTablet for tablet bd117d131d88434881e5458bd110dfa4 (DEFAULT_TABLE table=test-table [id=6e5c6d8e8b01435eb5864a7fd9f5968e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:07.214385 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd117d131d88434881e5458bd110dfa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.214794 2040 ts_tablet_manager.cc:1918] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:07.214852 2040 log.cc:1196] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/78b626f5aa9043dd9e92718778223721
I20260620 14:19:07.215103 2040 ts_tablet_manager.cc:1939] T 78b626f5aa9043dd9e92718778223721 P c8812f059b3e476ab29e64dc0177d1d2: Deleting consensus metadata
I20260620 14:19:07.215632 2038 ts_tablet_manager.cc:1918] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:07.215698 2038 log.cc:1196] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/78b626f5aa9043dd9e92718778223721
I20260620 14:19:07.215687 1495 catalog_manager.cc:4841] TS c8812f059b3e476ab29e64dc0177d1d2 (127.0.98.2:34205): tablet 78b626f5aa9043dd9e92718778223721 (table test-table [id=b11166c662cb4b10abe5ba430ff20592]) successfully deleted
I20260620 14:19:07.215705 1926 tablet_service.cc:1467] Processing CreateTablet for tablet bd117d131d88434881e5458bd110dfa4 (DEFAULT_TABLE table=test-table [id=6e5c6d8e8b01435eb5864a7fd9f5968e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:07.215801 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd117d131d88434881e5458bd110dfa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.215981 2039 ts_tablet_manager.cc:1918] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:07.216001 2038 ts_tablet_manager.cc:1939] T 78b626f5aa9043dd9e92718778223721 P 1622ef25ba9e44249c4f5843d6a3ec72: Deleting consensus metadata
I20260620 14:19:07.216027 2039 log.cc:1196] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/78b626f5aa9043dd9e92718778223721
I20260620 14:19:07.216207 2039 ts_tablet_manager.cc:1939] T 78b626f5aa9043dd9e92718778223721 P 1e6262240e13482aa9cbe916167aebf6: Deleting consensus metadata
I20260620 14:19:07.216642 1495 catalog_manager.cc:4841] TS 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3:39261): tablet 78b626f5aa9043dd9e92718778223721 (table test-table [id=b11166c662cb4b10abe5ba430ff20592]) successfully deleted
I20260620 14:19:07.217182 2015 tablet_bootstrap.cc:492] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Bootstrap starting.
I20260620 14:19:07.217296 1493 catalog_manager.cc:4841] TS 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629): tablet 78b626f5aa9043dd9e92718778223721 (table test-table [id=b11166c662cb4b10abe5ba430ff20592]) successfully deleted
I20260620 14:19:07.217562 2015 tablet_bootstrap.cc:654] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.217600 1646 tablet_service.cc:1467] Processing CreateTablet for tablet bd117d131d88434881e5458bd110dfa4 (DEFAULT_TABLE table=test-table [id=6e5c6d8e8b01435eb5864a7fd9f5968e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:07.217710 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd117d131d88434881e5458bd110dfa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.218142 2015 tablet_bootstrap.cc:492] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: No bootstrap required, opened a new log
I20260620 14:19:07.218180 2015 ts_tablet_manager.cc:1397] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:07.218333 2015 raft_consensus.cc:348] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.218390 2015 raft_consensus.cc:374] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.218405 2015 raft_consensus.cc:729] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6262240e13482aa9cbe916167aebf6, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.218468 2015 consensus_queue.cc:260] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.218556 2015 ts_tablet_manager.cc:1428] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.218897 2016 tablet_bootstrap.cc:492] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Bootstrap starting.
I20260620 14:19:07.219319 2016 tablet_bootstrap.cc:654] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.219818 2021 tablet_bootstrap.cc:492] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: Bootstrap starting.
I20260620 14:19:07.219902 2016 tablet_bootstrap.cc:492] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: No bootstrap required, opened a new log
I20260620 14:19:07.219933 2016 ts_tablet_manager.cc:1397] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:07.220031 2016 raft_consensus.cc:348] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.220075 2016 raft_consensus.cc:374] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.220088 2016 raft_consensus.cc:729] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1622ef25ba9e44249c4f5843d6a3ec72, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.220145 2016 consensus_queue.cc:260] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.220228 2016 ts_tablet_manager.cc:1428] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.220242 2021 tablet_bootstrap.cc:654] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.220846 2021 tablet_bootstrap.cc:492] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: No bootstrap required, opened a new log
I20260620 14:19:07.220891 2021 ts_tablet_manager.cc:1397] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:07.221019 2021 raft_consensus.cc:348] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.221086 2021 raft_consensus.cc:374] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.221107 2021 raft_consensus.cc:729] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8812f059b3e476ab29e64dc0177d1d2, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.221154 2021 consensus_queue.cc:260] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.221236 2021 ts_tablet_manager.cc:1428] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260620 14:19:07.225548 1713 tablet.cc:2369] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:07.252904 2019 raft_consensus.cc:482] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:07.253021 2019 raft_consensus.cc:504] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.253389 2019 leader_election.cc:288] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629), c8812f059b3e476ab29e64dc0177d1d2 (127.0.98.2:34205)
I20260620 14:19:07.257230 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.253721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33173 (local address 127.0.98.2:34205)
0620 14:19:07.253817 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.253819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.253920 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:19:07.254023 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.254024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.254035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.254064 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:07.254328 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.255065 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.256453 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.256808 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.256895 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.256909 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.257097 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.257138 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:19:07.257331 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.253599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37629 (local address 127.0.98.3:54441)
0620 14:19:07.253642 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.253655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.253698 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.254018 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.254019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.254021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:07.254088 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.254090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.255008 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.255009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.256315 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.256318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.257086 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.257088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.257152 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.257154 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.257207 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.257252 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":5}
I20260620 14:19:07.257344 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.253772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54441 (local address 127.0.98.1:37629)
0620 14:19:07.253851 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.253853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.253861 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:07.253888 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.253890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.253898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.253963 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:07.254180 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.254892 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.256411 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.256990 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.257065 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.257075 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.257226 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.257269 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:19:07.257477 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.253599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34205 (local address 127.0.98.3:33173)
0620 14:19:07.253861 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.253871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.253946 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.254162 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.254163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.254165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:07.254249 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.254250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.255154 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.255154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.256315 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.256318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.256912 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.256914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.256986 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.256989 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.257326 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.257367 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":149,"thread_start_us":61,"threads_started":1}
I20260620 14:19:07.257829 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd117d131d88434881e5458bd110dfa4" candidate_uuid: "1e6262240e13482aa9cbe916167aebf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" is_pre_election: true
I20260620 14:19:07.257938 1666 raft_consensus.cc:2454] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e6262240e13482aa9cbe916167aebf6 in term 0.
I20260620 14:19:07.257902 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd117d131d88434881e5458bd110dfa4" candidate_uuid: "1e6262240e13482aa9cbe916167aebf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8812f059b3e476ab29e64dc0177d1d2" is_pre_election: true
I20260620 14:19:07.258033 1806 raft_consensus.cc:2454] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e6262240e13482aa9cbe916167aebf6 in term 0.
I20260620 14:19:07.258673 1879 leader_election.cc:302] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6262240e13482aa9cbe916167aebf6, c8812f059b3e476ab29e64dc0177d1d2; no voters:
I20260620 14:19:07.258857 2019 raft_consensus.cc:2781] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:07.258913 2019 raft_consensus.cc:482] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:07.258936 2019 raft_consensus.cc:3037] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.259852 2019 raft_consensus.cc:504] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.259995 2019 leader_election.cc:288] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [CANDIDATE]: Term 1 election: Requested vote from peers 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629), c8812f059b3e476ab29e64dc0177d1d2 (127.0.98.2:34205)
I20260620 14:19:07.260229 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd117d131d88434881e5458bd110dfa4" candidate_uuid: "1e6262240e13482aa9cbe916167aebf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8812f059b3e476ab29e64dc0177d1d2"
I20260620 14:19:07.260322 1806 raft_consensus.cc:3037] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.260720 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd117d131d88434881e5458bd110dfa4" candidate_uuid: "1e6262240e13482aa9cbe916167aebf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622ef25ba9e44249c4f5843d6a3ec72"
I20260620 14:19:07.260802 1666 raft_consensus.cc:3037] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.261150 1806 raft_consensus.cc:2454] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e6262240e13482aa9cbe916167aebf6 in term 1.
I20260620 14:19:07.261334 1879 leader_election.cc:302] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6262240e13482aa9cbe916167aebf6, c8812f059b3e476ab29e64dc0177d1d2; no voters:
I20260620 14:19:07.261406 2019 raft_consensus.cc:2781] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:07.261529 2019 raft_consensus.cc:686] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 1 LEADER]: Becoming Leader. State: Replica: 1e6262240e13482aa9cbe916167aebf6, State: Running, Role: LEADER
I20260620 14:19:07.261591 1666 raft_consensus.cc:2454] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e6262240e13482aa9cbe916167aebf6 in term 1.
I20260620 14:19:07.261605 2019 consensus_queue.cc:237] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.262286 1508 catalog_manager.cc:5495] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1e6262240e13482aa9cbe916167aebf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } health_report { overall_health: HEALTHY } } }
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260620 14:19:07.286810 1993 tablet.cc:2369] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:07.287897 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.280368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.0.98.62:38145)
0620 14:19:07.280470 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.280472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.280618 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:19:07.280717 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.280718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.280727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.280806 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:07.281303 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.281824 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.282542 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.282727 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.282808 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.285197 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:07.285199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:07.285199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:07.285205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:07.286690 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.287156 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.287157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.287157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.287210 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.287422 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.287422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.287423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.287513 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:07.287608 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.287766 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.287836 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:19:07.288030 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.280320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38145 (local address 127.0.0.1:42198)
0620 14:19:07.280533 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.280546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.280637 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.280812 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.280813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.281165 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:07.281215 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.281218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.281907 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.281908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.282445 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.282446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.282808 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.282809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.282860 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.283440 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:07.283445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:07.285079 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:07.286797 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:07.286804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:07.286806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:07.287077 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:07.287282 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:07.287283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:07.287283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:07.287359 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:07.287600 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:07.287606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:07.287659 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.287748 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.287933 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":67,"thread_start_us":125,"threads_started":1}
I20260620 14:19:07.290036 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:19:07.290277 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:07.291811 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc03eea607d4ba1b54d02f7f077ff86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10446931cfd24e3e9ad5124c79dedaf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:07.291827 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc03eea607d4ba1b54d02f7f077ff86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10446931cfd24e3e9ad5124c79dedaf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:07.291821 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc03eea607d4ba1b54d02f7f077ff86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10446931cfd24e3e9ad5124c79dedaf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:07.291970 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc03eea607d4ba1b54d02f7f077ff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.291970 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc03eea607d4ba1b54d02f7f077ff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.291972 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc03eea607d4ba1b54d02f7f077ff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:07.293434 2015 tablet_bootstrap.cc:492] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: Bootstrap starting.
I20260620 14:19:07.293817 2015 tablet_bootstrap.cc:654] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.294185 2021 tablet_bootstrap.cc:492] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: Bootstrap starting.
I20260620 14:19:07.294737 2021 tablet_bootstrap.cc:654] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.295020 2016 tablet_bootstrap.cc:492] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: Bootstrap starting.
I20260620 14:19:07.295377 2016 tablet_bootstrap.cc:654] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:07.295917 2015 tablet_bootstrap.cc:492] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: No bootstrap required, opened a new log
I20260620 14:19:07.295967 2015 ts_tablet_manager.cc:1397] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:19:07.295998 2016 tablet_bootstrap.cc:492] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: No bootstrap required, opened a new log
I20260620 14:19:07.296031 2016 ts_tablet_manager.cc:1397] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:07.296110 2015 raft_consensus.cc:348] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.296139 2016 raft_consensus.cc:348] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.296185 2016 raft_consensus.cc:374] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.296180 2015 raft_consensus.cc:374] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.296198 2016 raft_consensus.cc:729] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1622ef25ba9e44249c4f5843d6a3ec72, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.296204 2015 raft_consensus.cc:729] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6262240e13482aa9cbe916167aebf6, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.296247 2015 consensus_queue.cc:260] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.296254 2016 consensus_queue.cc:260] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.296340 2015 ts_tablet_manager.cc:1428] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.296366 2016 ts_tablet_manager.cc:1428] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.296417 2045 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.296471 2020 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.296507 2045 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:07.296545 2020 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:07.296634 2021 tablet_bootstrap.cc:492] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: No bootstrap required, opened a new log
I20260620 14:19:07.296675 2021 ts_tablet_manager.cc:1397] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 14:19:07.296818 2021 raft_consensus.cc:348] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.296890 2021 raft_consensus.cc:374] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:07.296909 2021 raft_consensus.cc:729] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8812f059b3e476ab29e64dc0177d1d2, State: Initialized, Role: FOLLOWER
I20260620 14:19:07.296957 2021 consensus_queue.cc:260] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.297047 2021 ts_tablet_manager.cc:1428] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:07.297063 2025 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.297171 2025 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:07.486384 2025 raft_consensus.cc:482] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:07.486488 2025 raft_consensus.cc:504] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.486662 2025 leader_election.cc:288] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629), 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3:39261)
I20260620 14:19:07.486934 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc03eea607d4ba1b54d02f7f077ff86" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6262240e13482aa9cbe916167aebf6" is_pre_election: true
I20260620 14:19:07.486934 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc03eea607d4ba1b54d02f7f077ff86" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" is_pre_election: true
I20260620 14:19:07.487046 1946 raft_consensus.cc:2454] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 0.
I20260620 14:19:07.487046 1666 raft_consensus.cc:2454] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 0.
I20260620 14:19:07.487205 1737 leader_election.cc:302] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1622ef25ba9e44249c4f5843d6a3ec72, c8812f059b3e476ab29e64dc0177d1d2; no voters:
I20260620 14:19:07.487351 2025 raft_consensus.cc:2781] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:07.487414 2025 raft_consensus.cc:482] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:07.487445 2025 raft_consensus.cc:3037] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.488212 2025 raft_consensus.cc:504] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.488368 2025 leader_election.cc:288] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 election: Requested vote from peers 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629), 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3:39261)
I20260620 14:19:07.488529 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc03eea607d4ba1b54d02f7f077ff86" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6262240e13482aa9cbe916167aebf6"
I20260620 14:19:07.488601 1946 raft_consensus.cc:3037] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.488557 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc03eea607d4ba1b54d02f7f077ff86" candidate_uuid: "c8812f059b3e476ab29e64dc0177d1d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622ef25ba9e44249c4f5843d6a3ec72"
I20260620 14:19:07.488668 1666 raft_consensus.cc:3037] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:07.489253 1666 raft_consensus.cc:2454] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 1.
I20260620 14:19:07.489436 1946 raft_consensus.cc:2454] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8812f059b3e476ab29e64dc0177d1d2 in term 1.
I20260620 14:19:07.489441 1737 leader_election.cc:302] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1622ef25ba9e44249c4f5843d6a3ec72, c8812f059b3e476ab29e64dc0177d1d2; no voters:
I20260620 14:19:07.489542 2025 raft_consensus.cc:2781] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:07.489609 2025 raft_consensus.cc:686] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 LEADER]: Becoming Leader. State: Replica: c8812f059b3e476ab29e64dc0177d1d2, State: Running, Role: LEADER
I20260620 14:19:07.489674 2025 consensus_queue.cc:237] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } }
I20260620 14:19:07.489974 2033 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: TxnStatusTablet state changed. Reason: New leader c8812f059b3e476ab29e64dc0177d1d2. Latest consensus state: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.490077 2033 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:07.490236 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:19:07.490303 1509 catalog_manager.cc:5495] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8812f059b3e476ab29e64dc0177d1d2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } health_report { overall_health: UNKNOWN } } }
I20260620 14:19:07.490440 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:19:07.527485 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.524829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34205 (local address 127.0.0.1:59052)
0620 14:19:07.524902 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.524915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.524965 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.525132 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.525133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.525134 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:07.525177 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.525179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.526092 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.526093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.526663 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.526665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.527130 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.527131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.527178 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.527191 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.527353 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.527359 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.527393 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.527428 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:19:07.527498 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.524914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.0.98.2:34205)
0620 14:19:07.524967 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.524969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.524980 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:07.525033 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.525034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.525046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.525125 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:07.525245 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.525994 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.526749 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.527058 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.527131 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.527270 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.527303 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.527355 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.527397 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.527416 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:19:07.528936 1946 raft_consensus.cc:1261] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Refusing update from remote peer c8812f059b3e476ab29e64dc0177d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:07.529112 1666 raft_consensus.cc:1261] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Refusing update from remote peer c8812f059b3e476ab29e64dc0177d1d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:07.529174 2033 consensus_queue.cc:1035] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:07.529399 2033 consensus_queue.cc:1035] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:07.529830 2045 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: TxnStatusTablet state changed. Reason: New leader c8812f059b3e476ab29e64dc0177d1d2. Latest consensus state: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.529922 2045 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:07.529961 2020 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: TxnStatusTablet state changed. Reason: New leader c8812f059b3e476ab29e64dc0177d1d2. Latest consensus state: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.530053 2020 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:07.530620 2033 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.530700 2033 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:07.530791 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298929244276842496 to 7298929244118044672. Current Snapshot: MvccSnapshot[applied={T|T < 7298929244276842496}]
I20260620 14:19:07.530885 2020 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.530964 2020 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1622ef25ba9e44249c4f5843d6a3ec72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:07.531466 2025 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.531543 2025 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P c8812f059b3e476ab29e64dc0177d1d2: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:07.531946 2045 tablet_replica.cc:439] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8812f059b3e476ab29e64dc0177d1d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 } } peers { permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 } } peers { permanent_uuid: "1e6262240e13482aa9cbe916167aebf6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39261 } } }
I20260620 14:19:07.532035 2045 tablet_replica.cc:442] T 5cc03eea607d4ba1b54d02f7f077ff86 P 1e6262240e13482aa9cbe916167aebf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:07.535526 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.532460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.0.98.3:39261)
0620 14:19:07.532502 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.532504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.532510 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:19:07.532541 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.532542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.532567 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.532644 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:07.532802 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.533478 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.534510 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.534832 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.534888 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.535071 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.535187 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.535245 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.535443 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.535461 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:19:07.540350 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.537937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48157 (local address 127.0.98.2:34205)
0620 14:19:07.537974 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.537976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.537984 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:07.538022 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.538023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.538033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.538089 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:07.538178 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.538872 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.539576 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.539897 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.539972 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.540145 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.540176 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.540233 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.540267 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.540286 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:19:07.540373 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.537854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34205 (local address 127.0.98.3:48157)
0620 14:19:07.537919 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.537929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.537969 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.538093 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.538093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.538094 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:07.538134 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.538135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.538950 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.538950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.539476 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.539478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.539997 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.539998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.540054 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.540065 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.540232 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.540233 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.540262 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.540291 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:19:07.542543 1666 raft_consensus.cc:1261] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Refusing update from remote peer 1e6262240e13482aa9cbe916167aebf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:07.542605 1806 raft_consensus.cc:1261] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Refusing update from remote peer 1e6262240e13482aa9cbe916167aebf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:07.542740 2019 consensus_queue.cc:1035] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1622ef25ba9e44249c4f5843d6a3ec72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:07.542829 2045 consensus_queue.cc:1035] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8812f059b3e476ab29e64dc0177d1d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:07.543838 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298929244332863488 to 7298929243184074752. Current Snapshot: MvccSnapshot[applied={T|T < 7298929244332863488}]
I20260620 14:19:07.554280 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.547971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39261 (local address 127.0.98.2:37097)
0620 14:19:07.551678 (+ 3707us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:07.551691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:07.551758 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:07.551937 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:07.551938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:07.551940 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:07.551981 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.551983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.552817 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.552818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.553515 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:07.553518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.553954 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.553955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:07.553993 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.554005 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.554163 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.554165 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:07.554195 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:19:07.554228 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3667,"thread_start_us":77,"threads_started":1}
I20260620 14:19:07.554289 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.548004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37097 (local address 127.0.98.3:39261)
0620 14:19:07.548039 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.548041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.551813 (+ 3772us) server_negotiation.cc:407] Connection header received
0620 14:19:07.551838 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.551839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.551848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.551895 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:07.552041 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.552741 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.553591 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.553881 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.553950 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.554077 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:07.554108 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:07.554158 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.554206 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.554222 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:19:07.561650 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:07.561757 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:07.562364 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aae9b47c9a354979b93ec894b88c6e5d: Sending DeleteTablet for 3 replicas of tablet bd117d131d88434881e5458bd110dfa4
I20260620 14:19:07.562608 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet bd117d131d88434881e5458bd110dfa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59036
I20260620 14:19:07.562621 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet bd117d131d88434881e5458bd110dfa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60926
I20260620 14:19:07.562691 2040 tablet_replica.cc:330] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: stopping tablet replica
I20260620 14:19:07.562702 2039 tablet_replica.cc:330] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: stopping tablet replica
I20260620 14:19:07.562732 2040 raft_consensus.cc:2229] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:07.562747 2039 raft_consensus.cc:2229] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:07.562765 2040 raft_consensus.cc:2258] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:07.562822 2039 raft_consensus.cc:2258] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:07.563117 2039 ts_tablet_manager.cc:1905] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:07.563217 2040 ts_tablet_manager.cc:1905] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:07.563769 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet bd117d131d88434881e5458bd110dfa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57230
I20260620 14:19:07.563868 2038 tablet_replica.cc:330] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: stopping tablet replica
I20260620 14:19:07.563923 2038 raft_consensus.cc:2229] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:07.563966 2038 raft_consensus.cc:2258] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:07.564597 2038 ts_tablet_manager.cc:1905] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:07.565274 2040 ts_tablet_manager.cc:1918] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:07.565339 2040 log.cc:1196] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/bd117d131d88434881e5458bd110dfa4
I20260620 14:19:07.565584 2040 ts_tablet_manager.cc:1939] T bd117d131d88434881e5458bd110dfa4 P c8812f059b3e476ab29e64dc0177d1d2: Deleting consensus metadata
I20260620 14:19:07.566164 2038 ts_tablet_manager.cc:1918] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:07.566241 2038 log.cc:1196] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/bd117d131d88434881e5458bd110dfa4
I20260620 14:19:07.566452 2038 ts_tablet_manager.cc:1939] T bd117d131d88434881e5458bd110dfa4 P 1622ef25ba9e44249c4f5843d6a3ec72: Deleting consensus metadata
I20260620 14:19:07.566754 1495 catalog_manager.cc:4841] TS c8812f059b3e476ab29e64dc0177d1d2 (127.0.98.2:34205): tablet bd117d131d88434881e5458bd110dfa4 (table test-table [id=6e5c6d8e8b01435eb5864a7fd9f5968e]) successfully deleted
I20260620 14:19:07.566864 1493 catalog_manager.cc:4841] TS 1622ef25ba9e44249c4f5843d6a3ec72 (127.0.98.1:37629): tablet bd117d131d88434881e5458bd110dfa4 (table test-table [id=6e5c6d8e8b01435eb5864a7fd9f5968e]) successfully deleted
I20260620 14:19:07.568203 2039 ts_tablet_manager.cc:1918] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:07.568265 2039 log.cc:1196] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/bd117d131d88434881e5458bd110dfa4
I20260620 14:19:07.568476 2039 ts_tablet_manager.cc:1939] T bd117d131d88434881e5458bd110dfa4 P 1e6262240e13482aa9cbe916167aebf6: Deleting consensus metadata
I20260620 14:19:07.568854 1495 catalog_manager.cc:4841] TS 1e6262240e13482aa9cbe916167aebf6 (127.0.98.3:39261): tablet bd117d131d88434881e5458bd110dfa4 (table test-table [id=6e5c6d8e8b01435eb5864a7fd9f5968e]) successfully deleted
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:07.578994 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.567816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.0.98.1:37629)
0620 14:19:07.567856 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.567857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.568104 (+ 247us) server_negotiation.cc:407] Connection header received
0620 14:19:07.568204 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.568205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.568218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.568248 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:07.569184 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.569889 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.570984 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.571271 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.571351 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.575037 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:07.575049 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:07.575049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:07.575056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:07.577634 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.578060 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.578063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.578064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.578135 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.578362 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.578363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.578363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.578552 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:07.578713 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.578803 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.578885 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:19:07.579530 1706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57238
I20260620 14:19:07.588426 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:07.580581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.0.98.62:38145)
0620 14:19:07.580624 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:07.580626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:07.580852 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:19:07.580940 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:07.580941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:07.580954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:07.581022 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:07.581640 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.582419 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.583448 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:07.583827 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:07.583909 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:07.585289 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:07.585291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:07.585291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:07.585299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:07.586987 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.587460 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.587461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.587462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.587521 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:07.587803 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:07.587804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:07.587804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:07.587932 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:07.588009 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:07.588251 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:19:07.588330 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:19:07.589026 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42210
I20260620 14:19:07.589845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 1581
I20260620 14:19:07.596912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 1721
I20260620 14:19:07.605609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 1861
I20260620 14:19:07.610810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 1477
2026-06-20T14:19:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2932 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[2076](info): set up 2 sockets
Jun 20 14:19:07 dist-test-slave-h6n5 krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260620 14:19:08.651206 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:08 dist-test-slave-h6n5 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:08.770294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43781
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:08.850515 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:08.850716 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:08.850739 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:08.851924 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:08.851981 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:19:08.851997 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:08.852010 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:08.852022 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:08.852035 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:08.853453 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43781
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:08.853643 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:08.853816 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:08.858682 2089 server_base.cc:1034] running on GCE node
W20260620 14:19:08.858657 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:08.858656 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:08.858690 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:08.859174 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:08.859509 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:08.860680 2089 hybrid_clock.cc:648] HybridClock initialized: now 1781965148860670 us; error 29 us; skew 500 ppm
Jun 20 14:19:08 dist-test-slave-h6n5 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:08.863145 2089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:08.863662 2089 webserver.cc:460] Webserver started at http://127.0.98.62:38395/ using document root <none> and password file <none>
I20260620 14:19:08.863903 2089 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:08.863971 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:08.864132 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:08.865190 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "55e1d9219458429293212124bf05cde0"
format_stamp: "Formatted at 2026-06-20 14:19:08 on dist-test-slave-h6n5"
server_key: "da850ff9c025a766c1a2713b48bbf333"
server_key_iv: "5612c4fa70829a39f3e9ea5625bc80c0"
server_key_version: "encryptionkey@0"
I20260620 14:19:08.865561 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "55e1d9219458429293212124bf05cde0"
format_stamp: "Formatted at 2026-06-20 14:19:08 on dist-test-slave-h6n5"
server_key: "da850ff9c025a766c1a2713b48bbf333"
server_key_iv: "5612c4fa70829a39f3e9ea5625bc80c0"
server_key_version: "encryptionkey@0"
I20260620 14:19:08.866997 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:19:08.867821 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:08.868028 2089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:08.868126 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "55e1d9219458429293212124bf05cde0"
format_stamp: "Formatted at 2026-06-20 14:19:08 on dist-test-slave-h6n5"
server_key: "da850ff9c025a766c1a2713b48bbf333"
server_key_iv: "5612c4fa70829a39f3e9ea5625bc80c0"
server_key_version: "encryptionkey@0"
I20260620 14:19:08.868214 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:08.881685 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:08.883757 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:08.883916 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:08.888931 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43781
I20260620 14:19:08.889074 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43781 every 8 connection(s)
I20260620 14:19:08.889353 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:08.890031 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:08.892434 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Bootstrap starting.
I20260620 14:19:08.893059 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:08.893358 2167 log.cc:824] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:08.894093 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: No bootstrap required, opened a new log
I20260620 14:19:08.895023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 2089
I20260620 14:19:08.895107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:08.895294 392 external_mini_cluster.cc:1428] Setting key f0af25d3ea0f8d4ceb885b116291d919
I20260620 14:19:08.896226 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e1d9219458429293212124bf05cde0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43781 } }
I20260620 14:19:08.896420 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:08.896468 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e1d9219458429293212124bf05cde0, State: Initialized, Role: FOLLOWER
I20260620 14:19:08.896608 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e1d9219458429293212124bf05cde0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43781 } }
I20260620 14:19:08.896688 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:08.896727 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:08.896767 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:08.897315 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e1d9219458429293212124bf05cde0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43781 } }
I20260620 14:19:08.897434 2167 leader_election.cc:302] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e1d9219458429293212124bf05cde0; no voters:
I20260620 14:19:08.897599 2167 leader_election.cc:288] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:08.897639 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:08.897786 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [term 1 LEADER]: Becoming Leader. State: Replica: 55e1d9219458429293212124bf05cde0, State: Running, Role: LEADER
I20260620 14:19:08.897879 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:08.897874 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e1d9219458429293212124bf05cde0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43781 } }
I20260620 14:19:08.899121 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55e1d9219458429293212124bf05cde0. Latest consensus state: current_term: 1 leader_uuid: "55e1d9219458429293212124bf05cde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e1d9219458429293212124bf05cde0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43781 } } }
I20260620 14:19:08.899222 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:08.899325 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55e1d9219458429293212124bf05cde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e1d9219458429293212124bf05cde0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43781 } } }
I20260620 14:19:08.899395 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0 [sys.catalog]: This master's current role is: LEADER
W20260620 14:19:08.899967 2187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:19:08.900064 2187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:19:08.900162 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:08.900317 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:08.902546 2186 catalog_manager.cc:1347] Generated new cluster ID: 23b2fb6048be4867a9569abd3f600bc1
I20260620 14:19:08.902595 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:08 dist-test-slave-h6n5 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:08.907498 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:08.896137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.98.62:43781)
0620 14:19:08.896315 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:08.896323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:08.896348 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:19:08.896475 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:08.896483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:08.896543 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:08.896751 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:08.897363 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:08.898837 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:08.900050 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:08.900679 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:08.900787 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:08.903415 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:08.903419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:08.903419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:08.903430 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:08.905849 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:08.906274 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:08.906278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:08.906278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:08.906389 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:08.906617 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:08.906617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:08.906618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:08.906908 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:08.907052 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:08.907182 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:19:08.907307 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260620 14:19:08.918184 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:08.918730 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:08.921178 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Generated new TSK 0
I20260620 14:19:08.921360 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:08.947197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43781
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:09.027437 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:09.027657 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:09.027683 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:09.028829 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:09.028872 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:09.028892 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:19:09.028944 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:09.028971 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:09.030475 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:09.030706 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:09.030994 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:09.035863 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:09.035874 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:09.035874 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:09.036098 2193 server_base.cc:1034] running on GCE node
I20260620 14:19:09.036295 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:09.036532 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:09.037707 2193 hybrid_clock.cc:648] HybridClock initialized: now 1781965149037700 us; error 39 us; skew 500 ppm
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:09.040381 2193 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:09.040838 2193 webserver.cc:460] Webserver started at http://127.0.98.1:41391/ using document root <none> and password file <none>
I20260620 14:19:09.041050 2193 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:09.041095 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:09.041272 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:09.042281 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c2f224c84884a90a10534795943ca1f"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "6afd24d327bb3ca1bcf5c9fe2ecb91d8"
server_key_iv: "15432e56c5134d396a089286d13c650c"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.042635 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c2f224c84884a90a10534795943ca1f"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "6afd24d327bb3ca1bcf5c9fe2ecb91d8"
server_key_iv: "15432e56c5134d396a089286d13c650c"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.043951 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:19:09.044667 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.044839 2193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:09.044937 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "0c2f224c84884a90a10534795943ca1f"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "6afd24d327bb3ca1bcf5c9fe2ecb91d8"
server_key_iv: "15432e56c5134d396a089286d13c650c"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.045012 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:09.064266 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:09.066306 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:09.066475 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:09.067152 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:09.067202 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.067235 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:09.067281 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:09.074720 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36713
I20260620 14:19:09.074779 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36713 every 8 connection(s)
I20260620 14:19:09.075150 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:09.077301 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.067589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39477 (local address 127.0.98.62:43781)
0620 14:19:09.067644 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.067646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.067978 (+ 332us) server_negotiation.cc:407] Connection header received
0620 14:19:09.068692 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.068693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.068702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.068742 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.069674 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.070280 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.072066 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.072328 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.072420 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.074664 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.074666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.074666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.074675 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.076239 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.076593 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.076594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.076594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.076648 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.076822 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.076823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.076823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.076915 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.076962 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.077182 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.077241 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:19:09.077505 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.067587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43781 (local address 127.0.98.1:39477)
0620 14:19:09.067815 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.067871 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.068569 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.068845 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.068850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.069242 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:09.069547 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.069553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.070383 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.070385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.071951 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.071955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.072438 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.072439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.072589 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.073072 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:09.073080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:09.074552 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:09.076313 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.076318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.076319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.076527 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.076690 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.076690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.076690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.076772 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.076972 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:09.076976 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:09.077079 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.077159 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.077269 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260620 14:19:09.078009 2324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43781
I20260620 14:19:09.078088 2324 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:09.078267 2324 heartbeater.cc:511] Master 127.0.98.62:43781 requested a full tablet report, sending...
I20260620 14:19:09.078794 2120 ts_manager.cc:194] Registered new tserver with Master: 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713)
I20260620 14:19:09.079496 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39477
I20260620 14:19:09.082275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 2193
I20260620 14:19:09.082367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:09.082549 392 external_mini_cluster.cc:1428] Setting key 40d70ef90d91168b96dfe3d404e1bbf2
I20260620 14:19:09.084955 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.079029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43781 (local address 127.0.98.1:60373)
0620 14:19:09.079171 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.079181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.079276 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.079405 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.079406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.079623 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:09.079682 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.079684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.080310 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.080311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.080814 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.080815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.081198 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.081199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.081271 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.081701 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:09.081706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:09.081965 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:09.083906 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.083907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.083908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.084094 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.084332 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.084333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.084333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.084412 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.084716 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:09.084717 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:09.084752 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.084811 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.084887 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260620 14:19:09.085067 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.079088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60373 (local address 127.0.98.62:43781)
0620 14:19:09.079126 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.079128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.079248 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:19:09.079334 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.079335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.079345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.079399 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.079760 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.080218 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.080911 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.081115 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.081197 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.082072 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.082073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.082074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.082080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.083805 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.084183 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.084184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.084184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.084241 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.084491 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.084492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.084492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.084636 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.084708 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.084858 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.084944 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:09.106580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43781
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:09.185433 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:09.185693 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:09.185738 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:09.186892 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:09.186966 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:09.187000 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:19:09.187029 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:09.187070 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:09.188537 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:09.188814 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:09.189047 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:09.194073 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:09.194197 2333 server_base.cc:1034] running on GCE node
W20260620 14:19:09.194101 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:09.194056 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:09.194511 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:09.194728 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:09.195868 2333 hybrid_clock.cc:648] HybridClock initialized: now 1781965149195851 us; error 34 us; skew 500 ppm
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:09.198642 2333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:09.199172 2333 webserver.cc:460] Webserver started at http://127.0.98.2:33535/ using document root <none> and password file <none>
I20260620 14:19:09.199486 2333 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:09.199558 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:09.199723 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:09.200781 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5e957fae7d74418a19bde5cbbfc2716"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "c8a56b4a6241f9251444bfd9a0aee0f4"
server_key_iv: "3a568d1cba2ffdc4fd20c3d440a2f9c4"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.201148 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5e957fae7d74418a19bde5cbbfc2716"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "c8a56b4a6241f9251444bfd9a0aee0f4"
server_key_iv: "3a568d1cba2ffdc4fd20c3d440a2f9c4"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.202469 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:09.203238 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.203414 2333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:09.203511 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "d5e957fae7d74418a19bde5cbbfc2716"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "c8a56b4a6241f9251444bfd9a0aee0f4"
server_key_iv: "3a568d1cba2ffdc4fd20c3d440a2f9c4"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.203605 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:09.214701 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:09.216776 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:09.216943 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:09.217501 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:09.217561 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.217603 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:09.217623 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:09.224929 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39763
I20260620 14:19:09.225032 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39763 every 8 connection(s)
I20260620 14:19:09.225350 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:09.227854 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.217930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55415 (local address 127.0.98.62:43781)
0620 14:19:09.217995 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.217998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.218192 (+ 194us) server_negotiation.cc:407] Connection header received
0620 14:19:09.218727 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.218728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.218740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.218812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.219467 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.220029 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.221304 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.221506 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.221591 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.224755 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.224757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.224757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.224773 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.226370 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.226840 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.226841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.226841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.226900 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.227204 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.227205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.227205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.227317 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.227371 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.227683 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.227783 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:09.228024 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.217903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43781 (local address 127.0.98.2:55415)
0620 14:19:09.218058 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.218097 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.218625 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.218879 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.218882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.219183 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:09.219365 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.219370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.220143 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.220145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.221208 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.221211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.221602 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.221602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.221734 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.222292 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:09.222303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:09.224644 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:09.226466 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.226503 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.226505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.226729 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.227007 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.227008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.227009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.227121 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.227380 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:09.227385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:09.227505 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.227651 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.227823 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260620 14:19:09.228533 2464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43781
I20260620 14:19:09.228622 2464 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:09.228818 2464 heartbeater.cc:511] Master 127.0.98.62:43781 requested a full tablet report, sending...
I20260620 14:19:09.229195 2121 ts_manager.cc:194] Registered new tserver with Master: d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763)
I20260620 14:19:09.229583 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55415
I20260620 14:19:09.231199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 2333
I20260620 14:19:09.231312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:09.231500 392 external_mini_cluster.cc:1428] Setting key e28f4160486bd30f3e6e95f38a84cade
I20260620 14:19:09.236758 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.229703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43191 (local address 127.0.98.62:43781)
0620 14:19:09.229743 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.229745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.229816 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:19:09.229894 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.229895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.229906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.229956 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.230344 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.230825 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.232499 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.232817 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.232884 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.234037 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.234039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.234040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.234048 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.235493 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.235936 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.235937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.235937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.235986 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.236195 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.236196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.236196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.236328 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.236412 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.236586 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.236661 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:19:09.236771 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.229608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43781 (local address 127.0.98.2:43191)
0620 14:19:09.229762 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.229773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.229837 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.229979 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.229980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.230182 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:09.230246 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.230248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.230911 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.230912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.232415 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.232419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.232937 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.232939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.233026 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.233570 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:09.233576 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:09.233951 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:09.235608 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.235609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.235610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.235862 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.236053 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.236054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.236055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.236131 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.236406 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:09.236407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:09.236466 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.236550 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.236672 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:09.257995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43781
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:09.339673 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:09.339886 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:09.339912 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:09.341014 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:09.341046 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:09.341065 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:19:09.341078 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:09.341095 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:09.342480 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:09.342677 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:09.342857 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:09.347787 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:09.347816 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:09.347780 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:09.348023 2473 server_base.cc:1034] running on GCE node
I20260620 14:19:09.348253 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:09.348507 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:09.349669 2473 hybrid_clock.cc:648] HybridClock initialized: now 1781965149349666 us; error 37 us; skew 500 ppm
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:09.352244 2473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:09.352700 2473 webserver.cc:460] Webserver started at http://127.0.98.3:44167/ using document root <none> and password file <none>
I20260620 14:19:09.352944 2473 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:09.353015 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:09.353168 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:09.354251 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "bece832cb4f04af5ad601ff7a487d69f"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "8b97d14ea9a91bc927199b58222cecee"
server_key_iv: "42d45c063b2213871667b375e4b29c9d"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.354614 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bece832cb4f04af5ad601ff7a487d69f"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "8b97d14ea9a91bc927199b58222cecee"
server_key_iv: "42d45c063b2213871667b375e4b29c9d"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.356062 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:09.356873 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.357043 2473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:09.357159 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "bece832cb4f04af5ad601ff7a487d69f"
format_stamp: "Formatted at 2026-06-20 14:19:09 on dist-test-slave-h6n5"
server_key: "8b97d14ea9a91bc927199b58222cecee"
server_key_iv: "42d45c063b2213871667b375e4b29c9d"
server_key_version: "encryptionkey@0"
I20260620 14:19:09.357234 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:09.369701 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:09.371990 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:09.372160 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:09.372695 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:09.372757 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.372798 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:09.372828 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:09.380043 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35009
I20260620 14:19:09.380122 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35009 every 8 connection(s)
I20260620 14:19:09.380453 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:09.382709 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.373216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35885 (local address 127.0.98.62:43781)
0620 14:19:09.373294 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.373297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.373514 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:19:09.373991 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.373993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.374005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.374079 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.374908 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.375775 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.376830 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.377160 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.377234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.379831 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.379833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.379833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.379841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.381326 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.381738 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.381739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.381740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.381804 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.382057 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.382058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.382058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.382185 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.382262 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.382533 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.382613 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:19:09.382720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 2473
I20260620 14:19:09.382860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:09.382911 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.373201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43781 (local address 127.0.98.3:35885)
0620 14:19:09.373383 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.373423 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.373895 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.374135 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.374138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.374473 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:09.374796 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.374802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.375872 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.375874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.376734 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.376737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.377251 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.377252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.377401 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.377982 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:09.377993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:09.379734 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:09.381425 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.381431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.381434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.381661 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.381871 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.381872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.381873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.381986 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.382253 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:09.382259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:09.382400 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.382509 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.382678 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":119,"threads_started":1}
I20260620 14:19:09.383098 392 external_mini_cluster.cc:1428] Setting key a1bdfb64838331e30d33b1720806c6c4
I20260620 14:19:09.383446 2604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43781
I20260620 14:19:09.383599 2604 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:09.383854 2604 heartbeater.cc:511] Master 127.0.98.62:43781 requested a full tablet report, sending...
I20260620 14:19:09.384307 2120 ts_manager.cc:194] Registered new tserver with Master: bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009)
I20260620 14:19:09.384824 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35885
I20260620 14:19:09.390367 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.384343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43781 (local address 127.0.98.3:50551)
0620 14:19:09.384496 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.384507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.384559 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.385092 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.385093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.385320 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:09.385389 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.385390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.386135 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.386136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.386715 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.386717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.387072 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.387072 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.387129 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.387533 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:09.387537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:09.387895 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:09.389419 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.389420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.389420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.389586 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.389793 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.389794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.389794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.389854 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.390134 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:09.390134 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:09.390168 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.390220 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.390295 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260620 14:19:09.390383 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.384435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50551 (local address 127.0.98.62:43781)
0620 14:19:09.384978 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.384979 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.384986 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:09.385001 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.385001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.385008 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.385073 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.385473 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.386020 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.386795 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.386989 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.387059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.387979 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.387980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.387980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.387987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.389345 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.389655 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.389656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.389656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.389706 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.389947 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.389948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.389948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.390067 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.390131 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.390236 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.390292 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516}
I20260620 14:19:09.394896 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:09.418278 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.411476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.0.98.62:43781)
0620 14:19:09.411595 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.411597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.411610 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:19:09.411651 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.411652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.411662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.411715 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.412291 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.412825 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.413531 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.413724 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.413778 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.416001 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.416003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.416003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.416010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.417267 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.417592 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.417593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.417594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.417653 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.417818 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.417819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.417819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.417917 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.417991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.418095 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.418156 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:19:09.420027 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:09.421756 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:09.428009 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.424062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.0.98.3:35009)
0620 14:19:09.424366 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.424372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.424382 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:09.424424 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.424426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.424443 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.424742 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.424807 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.425536 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.426977 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.427519 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.427648 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.427718 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.427874 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.427915 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":80,"threads_started":1}
I20260620 14:19:09.428000 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.423969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35009 (local address 127.0.0.1:55014)
0620 14:19:09.424264 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.424275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.424332 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.424562 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.424565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.424567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.424636 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.424638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.425623 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.425624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.426893 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.426895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.427662 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.427663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.427739 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.427752 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.427858 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.427912 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":2,"thread_start_us":44,"threads_started":1}
I20260620 14:19:09.428186 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.423969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39763 (local address 127.0.0.1:39880)
0620 14:19:09.424133 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.424173 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.424235 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.424574 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.424575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.424577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.424658 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.424660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.425966 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.425967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.427034 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.427041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.427960 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.427961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.428021 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.428025 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.428056 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.428095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":32,"thread_start_us":67,"threads_started":1}
I20260620 14:19:09.428373 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.424367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.0.98.1:36713)
0620 14:19:09.424516 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.424520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.424681 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:19:09.424725 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.424727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.424743 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.424815 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.424986 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.426034 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.427096 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.427829 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.427962 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.428022 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.428239 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.428278 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260620 14:19:09.428496 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.424173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.0.98.2:39763)
0620 14:19:09.424436 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.424440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.424448 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:09.424484 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.424485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.424500 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.424575 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.424750 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.425878 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.427142 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.427871 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.428257 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.428318 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.428345 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.428383 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
I20260620 14:19:09.428305 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.424151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36713 (local address 127.0.0.1:37648)
0620 14:19:09.424535 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.424544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.424595 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.424874 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.424875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.424876 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.424913 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.424915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.426124 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.426125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.427014 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.427016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.428045 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.428046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.428151 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.428154 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.428210 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.428252 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":72,"threads_started":1}
I20260620 14:19:09.429114 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 7c67c31e1ea54421a94bbac4fd437b37 (DEFAULT_TABLE table=test-table [id=41608d5d188749dcb0a944f0617806fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:09.429275 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 7c67c31e1ea54421a94bbac4fd437b37 (DEFAULT_TABLE table=test-table [id=41608d5d188749dcb0a944f0617806fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:09.429370 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 7c67c31e1ea54421a94bbac4fd437b37 (DEFAULT_TABLE table=test-table [id=41608d5d188749dcb0a944f0617806fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:09.429533 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c67c31e1ea54421a94bbac4fd437b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.429557 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c67c31e1ea54421a94bbac4fd437b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.429711 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c67c31e1ea54421a94bbac4fd437b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.432668 2627 tablet_bootstrap.cc:492] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Bootstrap starting.
I20260620 14:19:09.433030 2629 tablet_bootstrap.cc:492] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Bootstrap starting.
I20260620 14:19:09.433194 2628 tablet_bootstrap.cc:492] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Bootstrap starting.
I20260620 14:19:09.433404 2627 tablet_bootstrap.cc:654] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.433707 2627 log.cc:824] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:09.433735 2629 tablet_bootstrap.cc:654] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.434049 2629 log.cc:824] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:09.434145 2628 tablet_bootstrap.cc:654] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.434463 2627 tablet_bootstrap.cc:492] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: No bootstrap required, opened a new log
I20260620 14:19:09.434445 2628 log.cc:824] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:09.434536 2627 ts_tablet_manager.cc:1397] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:09.434799 2629 tablet_bootstrap.cc:492] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: No bootstrap required, opened a new log
I20260620 14:19:09.434882 2629 ts_tablet_manager.cc:1397] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:09.435227 2628 tablet_bootstrap.cc:492] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: No bootstrap required, opened a new log
I20260620 14:19:09.435343 2628 ts_tablet_manager.cc:1397] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:09.436323 2627 raft_consensus.cc:348] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.436460 2627 raft_consensus.cc:374] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.436483 2627 raft_consensus.cc:729] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2f224c84884a90a10534795943ca1f, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.436650 2627 consensus_queue.cc:260] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.436933 2324 heartbeater.cc:503] Master 127.0.98.62:43781 was elected leader, sending a full tablet report...
I20260620 14:19:09.436910 2629 raft_consensus.cc:348] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.437070 2629 raft_consensus.cc:374] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.437117 2629 raft_consensus.cc:729] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e957fae7d74418a19bde5cbbfc2716, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.436928 2627 ts_tablet_manager.cc:1428] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:19:09.437266 2629 consensus_queue.cc:260] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.437556 2629 ts_tablet_manager.cc:1428] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:19:09.437666 2464 heartbeater.cc:503] Master 127.0.98.62:43781 was elected leader, sending a full tablet report...
I20260620 14:19:09.437686 2628 raft_consensus.cc:348] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.437887 2628 raft_consensus.cc:374] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.437933 2628 raft_consensus.cc:729] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.438048 2628 consensus_queue.cc:260] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.438297 2628 ts_tablet_manager.cc:1428] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:19:09.438400 2604 heartbeater.cc:503] Master 127.0.98.62:43781 was elected leader, sending a full tablet report...
W20260620 14:19:09.476073 2465 tablet.cc:2369] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:09.546720 2634 raft_consensus.cc:482] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:09.546837 2634 raft_consensus.cc:504] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.547295 2634 leader_election.cc:288] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009)
I20260620 14:19:09.550266 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.547489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35009 (local address 127.0.98.2:54445)
0620 14:19:09.547541 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.547553 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.547619 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.547780 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.547781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.547783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.547862 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.547864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.548630 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.548631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.549417 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.549419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.549994 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.549995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.550073 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.550077 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.550157 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.550196 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:19:09.550266 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.547578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54445 (local address 127.0.98.3:35009)
0620 14:19:09.547643 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.547645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.547654 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:09.547690 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.547691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.547701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.547773 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.547938 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.548529 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.549512 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.549898 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.549994 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.550004 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.550165 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.550198 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:09.550516 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.547578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44519 (local address 127.0.98.1:36713)
0620 14:19:09.547624 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.547627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.547918 (+ 291us) server_negotiation.cc:407] Connection header received
0620 14:19:09.547978 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.547979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.547992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.548055 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.548253 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.548807 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.549753 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.550122 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.550205 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.550213 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.550398 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.550451 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:09.550611 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c67c31e1ea54421a94bbac4fd437b37" candidate_uuid: "d5e957fae7d74418a19bde5cbbfc2716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bece832cb4f04af5ad601ff7a487d69f" is_pre_election: true
I20260620 14:19:09.550735 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.547492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36713 (local address 127.0.98.2:44519)
0620 14:19:09.547834 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.547846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.547923 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.548058 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.548059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.548060 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.548147 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.548149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.548884 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.548885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.549659 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.549661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.550206 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.550207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.550287 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.550290 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.550608 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.550645 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":182,"thread_start_us":108,"threads_started":1}
I20260620 14:19:09.550782 2558 raft_consensus.cc:2454] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5e957fae7d74418a19bde5cbbfc2716 in term 0.
I20260620 14:19:09.550993 2351 leader_election.cc:302] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bece832cb4f04af5ad601ff7a487d69f, d5e957fae7d74418a19bde5cbbfc2716; no voters:
I20260620 14:19:09.551012 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c67c31e1ea54421a94bbac4fd437b37" candidate_uuid: "d5e957fae7d74418a19bde5cbbfc2716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f" is_pre_election: true
I20260620 14:19:09.551113 2634 raft_consensus.cc:2781] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:09.551172 2634 raft_consensus.cc:482] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:09.551203 2634 raft_consensus.cc:3037] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.551192 2278 raft_consensus.cc:2454] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5e957fae7d74418a19bde5cbbfc2716 in term 0.
I20260620 14:19:09.552192 2634 raft_consensus.cc:504] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.552340 2634 leader_election.cc:288] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [CANDIDATE]: Term 1 election: Requested vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009)
I20260620 14:19:09.552479 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c67c31e1ea54421a94bbac4fd437b37" candidate_uuid: "d5e957fae7d74418a19bde5cbbfc2716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bece832cb4f04af5ad601ff7a487d69f"
I20260620 14:19:09.552565 2558 raft_consensus.cc:3037] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.553341 2558 raft_consensus.cc:2454] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5e957fae7d74418a19bde5cbbfc2716 in term 1.
I20260620 14:19:09.553464 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c67c31e1ea54421a94bbac4fd437b37" candidate_uuid: "d5e957fae7d74418a19bde5cbbfc2716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f"
I20260620 14:19:09.553519 2351 leader_election.cc:302] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bece832cb4f04af5ad601ff7a487d69f, d5e957fae7d74418a19bde5cbbfc2716; no voters:
I20260620 14:19:09.553548 2278 raft_consensus.cc:3037] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.553617 2634 raft_consensus.cc:2781] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:09.553782 2634 raft_consensus.cc:686] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 LEADER]: Becoming Leader. State: Replica: d5e957fae7d74418a19bde5cbbfc2716, State: Running, Role: LEADER
I20260620 14:19:09.553880 2634 consensus_queue.cc:237] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.554534 2278 raft_consensus.cc:2454] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5e957fae7d74418a19bde5cbbfc2716 in term 1.
I20260620 14:19:09.554796 2121 catalog_manager.cc:5495] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d5e957fae7d74418a19bde5cbbfc2716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } health_report { overall_health: UNKNOWN } } }
W20260620 14:19:09.575899 2325 tablet.cc:2369] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:19:09 dist-test-slave-h6n5 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:09.595430 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.588386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43781 (local address 127.0.0.1:36184)
0620 14:19:09.588604 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.588615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.588683 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.588830 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.588831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.589148 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:09.589196 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.589198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.589873 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.589874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.590412 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.590414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.590847 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.590849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.590913 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.591477 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:09.591482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:09.592950 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:09.594416 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.594421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.594424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.594621 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.594860 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:09.594861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:09.594861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:09.594926 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:09.595145 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:09.595151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:09.595191 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.595244 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.595335 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260620 14:19:09.595476 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.588458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.0.98.62:43781)
0620 14:19:09.588573 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.588575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.588693 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:19:09.588727 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.588728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.588737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.588790 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:09.589275 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.589786 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.590541 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.590751 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.590816 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.593046 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:09.593048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:09.593048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:09.593054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:09.594338 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.594684 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.594685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.594686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.594750 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:09.594988 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:09.594989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:09.594989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:09.595085 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:09.595135 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.595275 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.595381 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":80}
I20260620 14:19:09.597606 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36184:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:19:09.597841 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:09.599727 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 715d7fa35c6e42389250fa7cb628a3ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=012bbfd4dcb241abaa1854a1219c6902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:09.599740 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 715d7fa35c6e42389250fa7cb628a3ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=012bbfd4dcb241abaa1854a1219c6902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:09.599757 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 715d7fa35c6e42389250fa7cb628a3ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=012bbfd4dcb241abaa1854a1219c6902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:09.599917 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715d7fa35c6e42389250fa7cb628a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.599918 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715d7fa35c6e42389250fa7cb628a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.599928 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715d7fa35c6e42389250fa7cb628a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.601197 2629 tablet_bootstrap.cc:492] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: Bootstrap starting.
I20260620 14:19:09.601414 2628 tablet_bootstrap.cc:492] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: Bootstrap starting.
I20260620 14:19:09.601696 2629 tablet_bootstrap.cc:654] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.601997 2628 tablet_bootstrap.cc:654] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.602360 2627 tablet_bootstrap.cc:492] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: Bootstrap starting.
I20260620 14:19:09.602478 2629 tablet_bootstrap.cc:492] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: No bootstrap required, opened a new log
I20260620 14:19:09.602538 2629 ts_tablet_manager.cc:1397] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:09.602702 2628 tablet_bootstrap.cc:492] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: No bootstrap required, opened a new log
I20260620 14:19:09.602690 2629 raft_consensus.cc:348] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.602756 2629 raft_consensus.cc:374] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.602753 2628 ts_tablet_manager.cc:1397] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:09.602756 2627 tablet_bootstrap.cc:654] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.602770 2629 raft_consensus.cc:729] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e957fae7d74418a19bde5cbbfc2716, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.602831 2629 consensus_queue.cc:260] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.602926 2629 ts_tablet_manager.cc:1428] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.602923 2628 raft_consensus.cc:348] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.602978 2634 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.603018 2628 raft_consensus.cc:374] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.603044 2628 raft_consensus.cc:729] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.603041 2634 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:09.603088 2628 consensus_queue.cc:260] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.603188 2628 ts_tablet_manager.cc:1428] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.603240 2635 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.603343 2635 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:09.603994 2627 tablet_bootstrap.cc:492] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: No bootstrap required, opened a new log
I20260620 14:19:09.604038 2627 ts_tablet_manager.cc:1397] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:09.604143 2627 raft_consensus.cc:348] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.604197 2627 raft_consensus.cc:374] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.604213 2627 raft_consensus.cc:729] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2f224c84884a90a10534795943ca1f, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.604279 2627 consensus_queue.cc:260] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.604386 2627 ts_tablet_manager.cc:1428] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.604486 2633 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.604583 2633 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:19:09.631093 2605 tablet.cc:2369] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:09.680465 2633 raft_consensus.cc:482] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:09.680651 2633 raft_consensus.cc:504] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.681073 2633 leader_election.cc:288] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763), bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009)
I20260620 14:19:09.684521 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.681465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52645 (local address 127.0.98.3:35009)
0620 14:19:09.681510 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.681512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.681528 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:19:09.681564 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.681565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.681576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.681647 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.681854 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.682627 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.683718 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.684154 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.684236 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.684251 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.684407 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.684449 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:09.684546 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.681292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35009 (local address 127.0.98.1:52645)
0620 14:19:09.681439 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.681448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.681501 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.681677 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.681678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.681682 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.681771 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.681773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.682712 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.682713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.683630 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.683632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.684259 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.684260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.684328 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.684332 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.684402 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.684445 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":17,"thread_start_us":87,"threads_started":1}
I20260620 14:19:09.684687 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.681370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54761 (local address 127.0.98.2:39763)
0620 14:19:09.681441 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.681443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.681511 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:19:09.681587 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.681588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.681599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.681654 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.681853 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.682605 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.683928 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.684304 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.684389 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.684399 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.684585 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.684625 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:19:09.684701 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.681274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39763 (local address 127.0.98.1:54761)
0620 14:19:09.681429 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.681442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.681507 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.681677 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.681678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.681680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.681771 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.681774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.682735 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.682736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.683836 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.683843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.684403 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.684404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.684493 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.684497 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.684565 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.684607 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":7,"thread_start_us":105,"threads_started":1}
I20260620 14:19:09.685026 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715d7fa35c6e42389250fa7cb628a3ef" candidate_uuid: "0c2f224c84884a90a10534795943ca1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bece832cb4f04af5ad601ff7a487d69f" is_pre_election: true
I20260620 14:19:09.685120 2558 raft_consensus.cc:2454] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c2f224c84884a90a10534795943ca1f in term 0.
I20260620 14:19:09.685065 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715d7fa35c6e42389250fa7cb628a3ef" candidate_uuid: "0c2f224c84884a90a10534795943ca1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716" is_pre_election: true
I20260620 14:19:09.685217 2418 raft_consensus.cc:2454] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c2f224c84884a90a10534795943ca1f in term 0.
I20260620 14:19:09.685308 2211 leader_election.cc:302] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2f224c84884a90a10534795943ca1f, bece832cb4f04af5ad601ff7a487d69f; no voters:
I20260620 14:19:09.685489 2633 raft_consensus.cc:2781] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:09.685535 2633 raft_consensus.cc:482] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:09.685549 2633 raft_consensus.cc:3037] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.686255 2633 raft_consensus.cc:504] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.686405 2633 leader_election.cc:288] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [CANDIDATE]: Term 1 election: Requested vote from peers d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763), bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009)
I20260620 14:19:09.686554 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715d7fa35c6e42389250fa7cb628a3ef" candidate_uuid: "0c2f224c84884a90a10534795943ca1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716"
I20260620 14:19:09.686650 2418 raft_consensus.cc:3037] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.686614 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715d7fa35c6e42389250fa7cb628a3ef" candidate_uuid: "0c2f224c84884a90a10534795943ca1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bece832cb4f04af5ad601ff7a487d69f"
I20260620 14:19:09.686681 2558 raft_consensus.cc:3037] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.687412 2558 raft_consensus.cc:2454] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c2f224c84884a90a10534795943ca1f in term 1.
I20260620 14:19:09.687427 2418 raft_consensus.cc:2454] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c2f224c84884a90a10534795943ca1f in term 1.
I20260620 14:19:09.687682 2210 leader_election.cc:302] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2f224c84884a90a10534795943ca1f, d5e957fae7d74418a19bde5cbbfc2716; no voters:
I20260620 14:19:09.687778 2633 raft_consensus.cc:2781] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:09.687891 2633 raft_consensus.cc:686] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [term 1 LEADER]: Becoming Leader. State: Replica: 0c2f224c84884a90a10534795943ca1f, State: Running, Role: LEADER
I20260620 14:19:09.688014 2633 consensus_queue.cc:237] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.688437 2650 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: TxnStatusTablet state changed. Reason: New leader 0c2f224c84884a90a10534795943ca1f. Latest consensus state: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.688524 2650 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:09.688691 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:19:09.688843 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:19:09.688894 2120 catalog_manager.cc:5495] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c2f224c84884a90a10534795943ca1f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } health_report { overall_health: UNKNOWN } } }
I20260620 14:19:09.698129 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.695590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.0.98.1:36713)
0620 14:19:09.695671 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.695673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.695682 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:09.695757 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.695757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.695776 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.695840 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:09.695952 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.696444 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.697215 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.697495 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.697572 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.697766 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.697890 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.697984 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.698026 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.698049 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:19:09.698168 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.695508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36713 (local address 127.0.0.1:37654)
0620 14:19:09.695608 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.695622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.695672 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.695844 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.695845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.695846 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:09.695900 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.695902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.696532 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.696533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.697117 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.697119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.697571 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.697572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.697641 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.697659 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.697958 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.697967 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.698013 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.698065 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:19:09.700320 2558 raft_consensus.cc:1261] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Refusing update from remote peer 0c2f224c84884a90a10534795943ca1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:09.700346 2418 raft_consensus.cc:1261] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Refusing update from remote peer 0c2f224c84884a90a10534795943ca1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:09.700636 2633 consensus_queue.cc:1035] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:09.700773 2650 consensus_queue.cc:1035] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:09.702001 2634 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: TxnStatusTablet state changed. Reason: New leader 0c2f224c84884a90a10534795943ca1f. Latest consensus state: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.702113 2634 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:09.702049 2635 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: TxnStatusTablet state changed. Reason: New leader 0c2f224c84884a90a10534795943ca1f. Latest consensus state: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.702147 2635 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:09.703043 2650 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.703143 2650 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:09.703063 2633 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.703332 2633 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P 0c2f224c84884a90a10534795943ca1f: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:09.703637 2635 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.703645 2634 tablet_replica.cc:439] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c2f224c84884a90a10534795943ca1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } } }
I20260620 14:19:09.703722 2635 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P bece832cb4f04af5ad601ff7a487d69f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:09.703732 2634 tablet_replica.cc:442] T 715d7fa35c6e42389250fa7cb628a3ef P d5e957fae7d74418a19bde5cbbfc2716: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:09.704310 2657 mvcc.cc:205] Tried to move back new op lower bound from 7298929253169889280 to 7298929253122662400. Current Snapshot: MvccSnapshot[applied={T|T < 7298929253169889280}]
I20260620 14:19:09.704334 2655 mvcc.cc:205] Tried to move back new op lower bound from 7298929253169889280 to 7298929253122662400. Current Snapshot: MvccSnapshot[applied={T|T < 7298929253169889280}]
I20260620 14:19:09.704701 2656 mvcc.cc:205] Tried to move back new op lower bound from 7298929253169889280 to 7298929253122662400. Current Snapshot: MvccSnapshot[applied={T|T < 7298929253169889280}]
I20260620 14:19:09.709062 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.706531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.98.2:39763)
0620 14:19:09.706587 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.706589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.706602 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:19:09.706679 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.706680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.706694 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.706777 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:09.706884 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.707457 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.708261 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.708482 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.708565 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.708737 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.708844 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.708936 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.708956 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.708982 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:19:09.714040 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.711456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36713 (local address 127.0.98.2:42855)
0620 14:19:09.711509 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.711533 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.711596 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.711724 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.711724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.711725 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:09.711772 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.711773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.712631 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.712632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.713215 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.713217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.713683 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.713684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.713731 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.713744 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.713921 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.713922 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.713957 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.713989 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:19:09.714053 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.711523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42855 (local address 127.0.98.1:36713)
0620 14:19:09.711581 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.711584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.711594 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:09.711649 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.711650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.711661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.711723 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:09.711808 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.712531 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.713302 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.713605 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.713676 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.713829 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.713862 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.713914 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.713960 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.713979 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:19:09.717064 2278 raft_consensus.cc:1261] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Refusing update from remote peer d5e957fae7d74418a19bde5cbbfc2716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:09.717278 2558 raft_consensus.cc:1261] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Refusing update from remote peer d5e957fae7d74418a19bde5cbbfc2716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:09.717350 2637 consensus_queue.cc:1035] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:09.717456 2634 consensus_queue.cc:1035] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:09.726426 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.723433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49647 (local address 127.0.98.2:39763)
0620 14:19:09.723475 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.723477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.723656 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:19:09.723751 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.723751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.723763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.723812 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:09.723988 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.724477 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.725623 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.725808 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.725864 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.726140 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.726185 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.726242 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.726327 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.726349 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:19:09.726464 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.723382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39763 (local address 127.0.98.1:49647)
0620 14:19:09.723590 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.723602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.723682 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.723859 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.723860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.723862 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:09.723927 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.723929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.724554 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.724555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.725535 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.725538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.725881 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.725882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.725952 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.725966 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.726272 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.726275 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.726322 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.726369 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260620 14:19:09.742012 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:09.742149 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:09.743045 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Sending DeleteTablet for 3 replicas of tablet 7c67c31e1ea54421a94bbac4fd437b37
I20260620 14:19:09.743427 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c67c31e1ea54421a94bbac4fd437b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260620 14:19:09.743710 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c67c31e1ea54421a94bbac4fd437b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55014
I20260620 14:19:09.743891 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:09.744057 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:09.744060 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c67c31e1ea54421a94bbac4fd437b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37648
I20260620 14:19:09.744517 2674 tablet_replica.cc:330] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: stopping tablet replica
I20260620 14:19:09.744683 2674 raft_consensus.cc:2229] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:09.744848 2674 raft_consensus.cc:2258] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:09.745064 2676 tablet_replica.cc:330] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: stopping tablet replica
I20260620 14:19:09.745339 2398 tablet_service.cc:1467] Processing CreateTablet for tablet e43f1faf51034ed98b44bdcc1d236493 (DEFAULT_TABLE table=test-table [id=3b6ac471d11544da8a38188a1a3ec342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:09.745481 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43f1faf51034ed98b44bdcc1d236493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.745646 2674 ts_tablet_manager.cc:1905] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:09.747090 2629 tablet_bootstrap.cc:492] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: Bootstrap starting.
I20260620 14:19:09.747305 2676 raft_consensus.cc:2229] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:09.747445 2676 raft_consensus.cc:2258] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:09.747807 2258 tablet_service.cc:1467] Processing CreateTablet for tablet e43f1faf51034ed98b44bdcc1d236493 (DEFAULT_TABLE table=test-table [id=3b6ac471d11544da8a38188a1a3ec342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:09.747927 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43f1faf51034ed98b44bdcc1d236493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.748111 2676 ts_tablet_manager.cc:1905] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:09.749289 2627 tablet_bootstrap.cc:492] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: Bootstrap starting.
I20260620 14:19:09.749672 2627 tablet_bootstrap.cc:654] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.750103 2538 tablet_service.cc:1467] Processing CreateTablet for tablet e43f1faf51034ed98b44bdcc1d236493 (DEFAULT_TABLE table=test-table [id=3b6ac471d11544da8a38188a1a3ec342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:09.750221 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43f1faf51034ed98b44bdcc1d236493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:09.750409 2676 ts_tablet_manager.cc:1918] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:09.750520 2676 log.cc:1196] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/7c67c31e1ea54421a94bbac4fd437b37
I20260620 14:19:09.750880 2676 ts_tablet_manager.cc:1939] T 7c67c31e1ea54421a94bbac4fd437b37 P 0c2f224c84884a90a10534795943ca1f: Deleting consensus metadata
I20260620 14:19:09.751622 2627 tablet_bootstrap.cc:492] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: No bootstrap required, opened a new log
I20260620 14:19:09.751645 2628 tablet_bootstrap.cc:492] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: Bootstrap starting.
I20260620 14:19:09.751672 2627 ts_tablet_manager.cc:1397] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:09.751803 2627 raft_consensus.cc:348] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.751874 2627 raft_consensus.cc:374] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.751894 2627 raft_consensus.cc:729] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2f224c84884a90a10534795943ca1f, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.751937 2627 consensus_queue.cc:260] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.752029 2627 ts_tablet_manager.cc:1428] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.752036 2628 tablet_bootstrap.cc:654] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.752610 2106 catalog_manager.cc:4841] TS 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713): tablet 7c67c31e1ea54421a94bbac4fd437b37 (table test-table [id=41608d5d188749dcb0a944f0617806fb]) successfully deleted
I20260620 14:19:09.747520 2629 tablet_bootstrap.cc:654] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:09.749804 2674 ts_tablet_manager.cc:1918] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:09.753139 2674 log.cc:1196] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/7c67c31e1ea54421a94bbac4fd437b37
I20260620 14:19:09.753527 2674 ts_tablet_manager.cc:1939] T 7c67c31e1ea54421a94bbac4fd437b37 P d5e957fae7d74418a19bde5cbbfc2716: Deleting consensus metadata
I20260620 14:19:09.753988 2106 catalog_manager.cc:4841] TS d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763): tablet 7c67c31e1ea54421a94bbac4fd437b37 (table test-table [id=41608d5d188749dcb0a944f0617806fb]) successfully deleted
I20260620 14:19:09.754197 2675 tablet_replica.cc:330] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: stopping tablet replica
I20260620 14:19:09.754295 2628 tablet_bootstrap.cc:492] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: No bootstrap required, opened a new log
I20260620 14:19:09.754313 2675 raft_consensus.cc:2229] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:09.754343 2628 ts_tablet_manager.cc:1397] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:19:09.754380 2675 raft_consensus.cc:2258] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:09.754487 2628 raft_consensus.cc:348] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.754561 2628 raft_consensus.cc:374] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.754582 2628 raft_consensus.cc:729] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.754622 2628 consensus_queue.cc:260] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.754698 2628 ts_tablet_manager.cc:1428] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.754794 2629 tablet_bootstrap.cc:492] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: No bootstrap required, opened a new log
I20260620 14:19:09.754837 2629 ts_tablet_manager.cc:1397] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260620 14:19:09.754976 2629 raft_consensus.cc:348] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.755038 2629 raft_consensus.cc:374] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:09.755060 2629 raft_consensus.cc:729] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e957fae7d74418a19bde5cbbfc2716, State: Initialized, Role: FOLLOWER
I20260620 14:19:09.755102 2629 consensus_queue.cc:260] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.755184 2629 ts_tablet_manager.cc:1428] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:09.755916 2675 ts_tablet_manager.cc:1905] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:09.757526 2675 ts_tablet_manager.cc:1918] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:09.757602 2675 log.cc:1196] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/7c67c31e1ea54421a94bbac4fd437b37
I20260620 14:19:09.757854 2675 ts_tablet_manager.cc:1939] T 7c67c31e1ea54421a94bbac4fd437b37 P bece832cb4f04af5ad601ff7a487d69f: Deleting consensus metadata
I20260620 14:19:09.758284 2107 catalog_manager.cc:4841] TS bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009): tablet 7c67c31e1ea54421a94bbac4fd437b37 (table test-table [id=41608d5d188749dcb0a944f0617806fb]) successfully deleted
I20260620 14:19:09.770241 2635 raft_consensus.cc:482] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:09.770357 2635 raft_consensus.cc:504] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.770661 2635 leader_election.cc:288] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763)
I20260620 14:19:09.774766 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.770891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36713 (local address 127.0.98.3:54549)
0620 14:19:09.771098 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.771109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.771165 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.771449 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.771450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.771451 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.771539 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.771542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.772533 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.772534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.773728 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.773730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.774500 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.774501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.774580 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.774584 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.774631 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.774672 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260620 14:19:09.775223 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.771272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43839 (local address 127.0.98.2:39763)
0620 14:19:09.771352 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.771354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.771363 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:09.771388 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.771388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.771397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.771446 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.772168 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.772917 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.774304 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.774872 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.774915 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.774924 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.775130 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.775160 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:19:09.775247 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.770941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54549 (local address 127.0.98.1:36713)
0620 14:19:09.770998 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.771000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.771308 (+ 308us) server_negotiation.cc:407] Connection header received
0620 14:19:09.771338 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.771339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.771349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.771401 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:09.771657 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.772403 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.773846 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.774389 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.774457 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.774468 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.775148 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.775183 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:19:09.775408 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f1faf51034ed98b44bdcc1d236493" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f" is_pre_election: true
I20260620 14:19:09.775460 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.770977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39763 (local address 127.0.98.3:43839)
0620 14:19:09.771099 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.771109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.771158 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.771492 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.771492 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.771494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:09.771954 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.771957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.773019 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.773020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.774200 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.774202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.774981 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.774982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.775052 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.775056 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.775344 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.775383 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260620 14:19:09.775490 2278 raft_consensus.cc:2454] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 0.
I20260620 14:19:09.775717 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f1faf51034ed98b44bdcc1d236493" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716" is_pre_election: true
I20260620 14:19:09.775799 2418 raft_consensus.cc:2454] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 0.
I20260620 14:19:09.775982 2490 leader_election.cc:302] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2f224c84884a90a10534795943ca1f, bece832cb4f04af5ad601ff7a487d69f; no voters:
I20260620 14:19:09.776145 2635 raft_consensus.cc:2781] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:09.776176 2635 raft_consensus.cc:482] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:09.776194 2635 raft_consensus.cc:3037] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.777060 2635 raft_consensus.cc:504] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.777199 2635 leader_election.cc:288] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 election: Requested vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763)
I20260620 14:19:09.777390 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f1faf51034ed98b44bdcc1d236493" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f"
I20260620 14:19:09.777403 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f1faf51034ed98b44bdcc1d236493" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716"
I20260620 14:19:09.777463 2278 raft_consensus.cc:3037] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.777468 2418 raft_consensus.cc:3037] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:09.778223 2418 raft_consensus.cc:2454] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 1.
I20260620 14:19:09.778265 2278 raft_consensus.cc:2454] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 1.
I20260620 14:19:09.778410 2490 leader_election.cc:302] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2f224c84884a90a10534795943ca1f, bece832cb4f04af5ad601ff7a487d69f; no voters:
I20260620 14:19:09.778533 2635 raft_consensus.cc:2781] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:09.778653 2635 raft_consensus.cc:686] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 1 LEADER]: Becoming Leader. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Running, Role: LEADER
I20260620 14:19:09.778716 2635 consensus_queue.cc:237] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:09.779359 2121 catalog_manager.cc:5495] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f reported cstate change: term changed from 0 to 1, leader changed from <none> to bece832cb4f04af5ad601ff7a487d69f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bece832cb4f04af5ad601ff7a487d69f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } health_report { overall_health: HEALTHY } } }
I20260620 14:19:09.789826 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.786584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.0.98.3:35009)
0620 14:19:09.786726 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.786728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.786735 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:09.786770 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.786771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.786791 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.786868 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:09.787044 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.787805 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.788902 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.789201 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.789283 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.789453 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.789585 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.789669 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.789724 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.789747 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 14:19:09.796802 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.792958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34489 (local address 127.0.98.1:36713)
0620 14:19:09.793258 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.793261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.793282 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:19:09.793332 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.793334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.793350 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.793468 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:09.793680 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.794570 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.795878 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.796222 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.796304 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.796519 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.796559 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.796614 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.796691 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.796714 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:19:09.796826 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.792744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36713 (local address 127.0.98.3:34489)
0620 14:19:09.792827 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.792839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.792879 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.793500 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.793501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.793503 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:09.793579 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.793581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.794689 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.794690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.795775 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.795778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.796322 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.796323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.796391 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.796416 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.796634 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.796636 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.796690 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.796740 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:19:09.799602 2418 raft_consensus.cc:1261] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Refusing update from remote peer bece832cb4f04af5ad601ff7a487d69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:09.799665 2278 raft_consensus.cc:1261] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Refusing update from remote peer bece832cb4f04af5ad601ff7a487d69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:09.799897 2635 consensus_queue.cc:1035] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:09.800064 2635 consensus_queue.cc:1035] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:09.810225 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.806588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35009 (local address 127.0.98.1:45171)
0620 14:19:09.806697 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:09.806709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:09.806745 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:09.807044 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:09.807045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:09.807047 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:09.807092 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.807094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.807999 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.808000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.808908 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:09.808911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.809799 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.809800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:09.809857 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.809864 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.810045 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.810047 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:09.810086 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 14:19:09.810136 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 14:19:09.810694 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:09.806827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45171 (local address 127.0.98.3:35009)
0620 14:19:09.806896 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:09.806898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:09.806904 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:19:09.806927 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:09.806928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:09.806937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:09.806984 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:09.807158 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.807906 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.809355 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:09.809663 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:09.809744 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:09.809939 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:09.809973 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:09.810015 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:09.810602 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 14:19:09.810624 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:19:09.810983 2656 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:19:11.332638 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:11.332743 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:11.333611 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Sending DeleteTablet for 3 replicas of tablet e43f1faf51034ed98b44bdcc1d236493
I20260620 14:19:11.333877 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet e43f1faf51034ed98b44bdcc1d236493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37648
I20260620 14:19:11.333930 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet e43f1faf51034ed98b44bdcc1d236493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260620 14:19:11.334111 2711 tablet_replica.cc:330] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: stopping tablet replica
I20260620 14:19:11.334178 2711 raft_consensus.cc:2229] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:11.334226 2711 raft_consensus.cc:2258] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:11.334494 2711 ts_tablet_manager.cc:1905] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:11.335489 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet e43f1faf51034ed98b44bdcc1d236493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55014
I20260620 14:19:11.336052 2711 ts_tablet_manager.cc:1918] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:19:11.336127 2711 log.cc:1196] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/e43f1faf51034ed98b44bdcc1d236493
I20260620 14:19:11.336337 2711 ts_tablet_manager.cc:1939] T e43f1faf51034ed98b44bdcc1d236493 P 0c2f224c84884a90a10534795943ca1f: Deleting consensus metadata
I20260620 14:19:11.336659 2106 catalog_manager.cc:4841] TS 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713): tablet e43f1faf51034ed98b44bdcc1d236493 (table test-table [id=3b6ac471d11544da8a38188a1a3ec342]) successfully deleted
I20260620 14:19:11.336856 2714 tablet_replica.cc:330] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: stopping tablet replica
I20260620 14:19:11.336930 2714 raft_consensus.cc:2229] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:11.337034 2714 raft_consensus.cc:2258] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:11.337163 2712 tablet_replica.cc:330] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: stopping tablet replica
I20260620 14:19:11.337229 2712 raft_consensus.cc:2229] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:11.337277 2712 raft_consensus.cc:2258] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:11.337261 2714 ts_tablet_manager.cc:1905] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:11.337543 2712 ts_tablet_manager.cc:1905] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:11.338595 2714 ts_tablet_manager.cc:1918] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:19:11.338673 2714 log.cc:1196] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/e43f1faf51034ed98b44bdcc1d236493
I20260620 14:19:11.338904 2714 ts_tablet_manager.cc:1939] T e43f1faf51034ed98b44bdcc1d236493 P bece832cb4f04af5ad601ff7a487d69f: Deleting consensus metadata
I20260620 14:19:11.339437 2712 ts_tablet_manager.cc:1918] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:19:11.339500 2712 log.cc:1196] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/e43f1faf51034ed98b44bdcc1d236493
I20260620 14:19:11.339491 2107 catalog_manager.cc:4841] TS bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009): tablet e43f1faf51034ed98b44bdcc1d236493 (table test-table [id=3b6ac471d11544da8a38188a1a3ec342]) successfully deleted
I20260620 14:19:11.339735 2712 ts_tablet_manager.cc:1939] T e43f1faf51034ed98b44bdcc1d236493 P d5e957fae7d74418a19bde5cbbfc2716: Deleting consensus metadata
I20260620 14:19:11.340058 2106 catalog_manager.cc:4841] TS d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763): tablet e43f1faf51034ed98b44bdcc1d236493 (table test-table [id=3b6ac471d11544da8a38188a1a3ec342]) successfully deleted
Jun 20 14:19:11 dist-test-slave-h6n5 krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965151, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:19:11 dist-test-slave-h6n5 krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965151, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:11.358067 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:11.350533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.0.98.62:43781)
0620 14:19:11.350780 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:11.350782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:11.350793 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:11.350906 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:11.350907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:11.350925 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:11.351004 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:11.351621 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:11.352136 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:11.352897 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:11.353103 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:11.353227 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:11.355499 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:11.355500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:11.355501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:11.355508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:11.356871 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:11.357224 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:11.357225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:11.357226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:11.357291 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:11.357489 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:11.357490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:11.357490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:11.357616 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:11.357730 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:11.357821 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:19:11.357912 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":69,"threads_started":1}
I20260620 14:19:11.359522 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:11.359814 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:11.361265 2538 tablet_service.cc:1467] Processing CreateTablet for tablet bdd1623426f1486a9f213b45fd458175 (DEFAULT_TABLE table=test-table [id=edba8bb2f4814f4c88bb152b5c4a91ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:11.361294 2398 tablet_service.cc:1467] Processing CreateTablet for tablet bdd1623426f1486a9f213b45fd458175 (DEFAULT_TABLE table=test-table [id=edba8bb2f4814f4c88bb152b5c4a91ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:11.361289 2258 tablet_service.cc:1467] Processing CreateTablet for tablet bdd1623426f1486a9f213b45fd458175 (DEFAULT_TABLE table=test-table [id=edba8bb2f4814f4c88bb152b5c4a91ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:11.361413 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd1623426f1486a9f213b45fd458175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:11.361413 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd1623426f1486a9f213b45fd458175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:11.361442 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd1623426f1486a9f213b45fd458175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:11.362485 2722 tablet_bootstrap.cc:492] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: Bootstrap starting.
I20260620 14:19:11.362820 2724 tablet_bootstrap.cc:492] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: Bootstrap starting.
I20260620 14:19:11.363085 2722 tablet_bootstrap.cc:654] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:11.363207 2724 tablet_bootstrap.cc:654] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:11.363509 2723 tablet_bootstrap.cc:492] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: Bootstrap starting.
I20260620 14:19:11.363706 2722 tablet_bootstrap.cc:492] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: No bootstrap required, opened a new log
I20260620 14:19:11.363767 2722 ts_tablet_manager.cc:1397] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:11.363803 2724 tablet_bootstrap.cc:492] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: No bootstrap required, opened a new log
I20260620 14:19:11.363850 2724 ts_tablet_manager.cc:1397] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:11.363885 2722 raft_consensus.cc:348] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.363950 2723 tablet_bootstrap.cc:654] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:11.363965 2722 raft_consensus.cc:374] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:11.363994 2722 raft_consensus.cc:729] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2f224c84884a90a10534795943ca1f, State: Initialized, Role: FOLLOWER
I20260620 14:19:11.363950 2724 raft_consensus.cc:348] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.364025 2724 raft_consensus.cc:374] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:11.364054 2724 raft_consensus.cc:729] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e957fae7d74418a19bde5cbbfc2716, State: Initialized, Role: FOLLOWER
I20260620 14:19:11.364045 2722 consensus_queue.cc:260] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.364120 2724 consensus_queue.cc:260] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.364236 2722 ts_tablet_manager.cc:1428] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:11.364298 2724 ts_tablet_manager.cc:1428] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:11.364519 2723 tablet_bootstrap.cc:492] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: No bootstrap required, opened a new log
I20260620 14:19:11.364574 2723 ts_tablet_manager.cc:1397] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:11.364708 2723 raft_consensus.cc:348] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.364790 2723 raft_consensus.cc:374] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:11.364820 2723 raft_consensus.cc:729] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Initialized, Role: FOLLOWER
I20260620 14:19:11.364863 2723 consensus_queue.cc:260] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.364969 2723 ts_tablet_manager.cc:1428] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:11.424278 2705 raft_consensus.cc:482] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:11.424381 2705 raft_consensus.cc:504] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.424602 2705 leader_election.cc:288] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763)
I20260620 14:19:11.424810 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd1623426f1486a9f213b45fd458175" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f" is_pre_election: true
I20260620 14:19:11.424885 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd1623426f1486a9f213b45fd458175" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716" is_pre_election: true
I20260620 14:19:11.424939 2278 raft_consensus.cc:2454] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 0.
I20260620 14:19:11.424968 2418 raft_consensus.cc:2454] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 0.
I20260620 14:19:11.425140 2490 leader_election.cc:302] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bece832cb4f04af5ad601ff7a487d69f, d5e957fae7d74418a19bde5cbbfc2716; no voters:
I20260620 14:19:11.425244 2705 raft_consensus.cc:2781] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:11.425287 2705 raft_consensus.cc:482] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:11.425321 2705 raft_consensus.cc:3037] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:11.425997 2705 raft_consensus.cc:504] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.426126 2705 leader_election.cc:288] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 election: Requested vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763)
I20260620 14:19:11.426282 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd1623426f1486a9f213b45fd458175" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716"
I20260620 14:19:11.426357 2418 raft_consensus.cc:3037] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:11.426294 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd1623426f1486a9f213b45fd458175" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f"
I20260620 14:19:11.426398 2278 raft_consensus.cc:3037] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:11.426944 2418 raft_consensus.cc:2454] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 1.
I20260620 14:19:11.427088 2278 raft_consensus.cc:2454] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 1.
I20260620 14:19:11.427179 2490 leader_election.cc:302] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bece832cb4f04af5ad601ff7a487d69f, d5e957fae7d74418a19bde5cbbfc2716; no voters:
I20260620 14:19:11.427316 2705 raft_consensus.cc:2781] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:11.427381 2705 raft_consensus.cc:686] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 1 LEADER]: Becoming Leader. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Running, Role: LEADER
I20260620 14:19:11.427448 2705 consensus_queue.cc:237] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.428128 2120 catalog_manager.cc:5495] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f reported cstate change: term changed from 0 to 1, leader changed from <none> to bece832cb4f04af5ad601ff7a487d69f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bece832cb4f04af5ad601ff7a487d69f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } health_report { overall_health: HEALTHY } } }
I20260620 14:19:11.443185 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:11.439461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.0.98.3:35009)
0620 14:19:11.441020 (+ 1559us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:11.441022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:11.441030 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:11.441051 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:11.441051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:11.441061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:11.441131 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:11.441328 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:11.441811 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:11.442610 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:11.442823 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:11.442874 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:11.442994 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:11.443017 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:11.443051 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:11.443104 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:19:11.443120 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1529,"thread_start_us":112,"threads_started":1}
I20260620 14:19:11.445622 2418 raft_consensus.cc:1261] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Refusing update from remote peer bece832cb4f04af5ad601ff7a487d69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:11.445859 2278 raft_consensus.cc:1261] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Refusing update from remote peer bece832cb4f04af5ad601ff7a487d69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:11.445844 2687 consensus_queue.cc:1035] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:11.446074 2687 consensus_queue.cc:1035] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:11.456478 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:11.456584 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:11.457188 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Sending DeleteTablet for 3 replicas of tablet bdd1623426f1486a9f213b45fd458175
I20260620 14:19:11.457468 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd1623426f1486a9f213b45fd458175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55014
I20260620 14:19:11.457459 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd1623426f1486a9f213b45fd458175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260620 14:19:11.457588 2712 tablet_replica.cc:330] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: stopping tablet replica
I20260620 14:19:11.457571 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd1623426f1486a9f213b45fd458175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37648
I20260620 14:19:11.457661 2711 tablet_replica.cc:330] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: stopping tablet replica
I20260620 14:19:11.457662 2712 raft_consensus.cc:2229] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:11.457712 2712 raft_consensus.cc:2258] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:11.457717 2711 raft_consensus.cc:2229] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:11.457762 2711 raft_consensus.cc:2258] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:11.457818 2714 tablet_replica.cc:330] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: stopping tablet replica
I20260620 14:19:11.457870 2714 raft_consensus.cc:2229] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:11.457924 2714 raft_consensus.cc:2258] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:11.458010 2712 ts_tablet_manager.cc:1905] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:11.458240 2711 ts_tablet_manager.cc:1905] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:11.458197 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:11.458325 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:11.459216 2714 ts_tablet_manager.cc:1905] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:11.461088 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 9da28e276fe04f67ae87aeb4c42af950 (DEFAULT_TABLE table=test-table [id=f6255f58f7d7417b98a500d8727bb94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:11.461129 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 9da28e276fe04f67ae87aeb4c42af950 (DEFAULT_TABLE table=test-table [id=f6255f58f7d7417b98a500d8727bb94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:11.461218 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 9da28e276fe04f67ae87aeb4c42af950 (DEFAULT_TABLE table=test-table [id=f6255f58f7d7417b98a500d8727bb94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:11.473002 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da28e276fe04f67ae87aeb4c42af950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:11.473039 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da28e276fe04f67ae87aeb4c42af950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:11.473006 2711 ts_tablet_manager.cc:1918] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:11.473042 2712 ts_tablet_manager.cc:1918] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:11.473261 2712 log.cc:1196] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/bdd1623426f1486a9f213b45fd458175
I20260620 14:19:11.473258 2711 log.cc:1196] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/bdd1623426f1486a9f213b45fd458175
I20260620 14:19:11.473505 2712 ts_tablet_manager.cc:1939] T bdd1623426f1486a9f213b45fd458175 P d5e957fae7d74418a19bde5cbbfc2716: Deleting consensus metadata
I20260620 14:19:11.473507 2711 ts_tablet_manager.cc:1939] T bdd1623426f1486a9f213b45fd458175 P 0c2f224c84884a90a10534795943ca1f: Deleting consensus metadata
I20260620 14:19:11.474318 2724 tablet_bootstrap.cc:492] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: Bootstrap starting.
I20260620 14:19:11.474618 2722 tablet_bootstrap.cc:492] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f: Bootstrap starting.
I20260620 14:19:11.474709 2724 tablet_bootstrap.cc:654] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:11.475016 2722 tablet_bootstrap.cc:654] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:11.475428 2724 tablet_bootstrap.cc:492] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: No bootstrap required, opened a new log
I20260620 14:19:11.475507 2724 ts_tablet_manager.cc:1397] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:11.475622 2722 tablet_bootstrap.cc:492] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f: No bootstrap required, opened a new log
I20260620 14:19:11.475669 2722 ts_tablet_manager.cc:1397] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:11.475685 2724 raft_consensus.cc:348] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.475775 2724 raft_consensus.cc:374] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:11.475795 2724 raft_consensus.cc:729] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e957fae7d74418a19bde5cbbfc2716, State: Initialized, Role: FOLLOWER
I20260620 14:19:11.475808 2722 raft_consensus.cc:348] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.475871 2722 raft_consensus.cc:374] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:11.475847 2724 consensus_queue.cc:260] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.475893 2722 raft_consensus.cc:729] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2f224c84884a90a10534795943ca1f, State: Initialized, Role: FOLLOWER
I20260620 14:19:11.475945 2724 ts_tablet_manager.cc:1428] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:11.475939 2722 consensus_queue.cc:260] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.476032 2722 ts_tablet_manager.cc:1428] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:11.483592 2106 catalog_manager.cc:4841] TS d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763): tablet bdd1623426f1486a9f213b45fd458175 (table test-table [id=edba8bb2f4814f4c88bb152b5c4a91ae]) successfully deleted
I20260620 14:19:11.483734 2106 catalog_manager.cc:4841] TS 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713): tablet bdd1623426f1486a9f213b45fd458175 (table test-table [id=edba8bb2f4814f4c88bb152b5c4a91ae]) successfully deleted
I20260620 14:19:11.484211 2714 ts_tablet_manager.cc:1918] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:11.484229 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da28e276fe04f67ae87aeb4c42af950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:11.484304 2714 log.cc:1196] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/bdd1623426f1486a9f213b45fd458175
I20260620 14:19:11.484546 2714 ts_tablet_manager.cc:1939] T bdd1623426f1486a9f213b45fd458175 P bece832cb4f04af5ad601ff7a487d69f: Deleting consensus metadata
I20260620 14:19:11.485846 2107 catalog_manager.cc:4841] TS bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009): tablet bdd1623426f1486a9f213b45fd458175 (table test-table [id=edba8bb2f4814f4c88bb152b5c4a91ae]) successfully deleted
I20260620 14:19:11.485903 2723 tablet_bootstrap.cc:492] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: Bootstrap starting.
I20260620 14:19:11.486253 2723 tablet_bootstrap.cc:654] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:11.486860 2723 tablet_bootstrap.cc:492] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: No bootstrap required, opened a new log
I20260620 14:19:11.486945 2723 ts_tablet_manager.cc:1397] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:11.487136 2723 raft_consensus.cc:348] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.487233 2723 raft_consensus.cc:374] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:11.487265 2723 raft_consensus.cc:729] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Initialized, Role: FOLLOWER
I20260620 14:19:11.487318 2723 consensus_queue.cc:260] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.487440 2723 ts_tablet_manager.cc:1428] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:11.661981 2702 raft_consensus.cc:482] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:11.662097 2702 raft_consensus.cc:504] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.662237 2702 leader_election.cc:288] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763)
I20260620 14:19:11.662496 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da28e276fe04f67ae87aeb4c42af950" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f" is_pre_election: true
I20260620 14:19:11.662592 2278 raft_consensus.cc:2454] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 0.
I20260620 14:19:11.662547 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da28e276fe04f67ae87aeb4c42af950" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716" is_pre_election: true
I20260620 14:19:11.662659 2418 raft_consensus.cc:2454] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 0.
I20260620 14:19:11.662752 2490 leader_election.cc:302] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2f224c84884a90a10534795943ca1f, bece832cb4f04af5ad601ff7a487d69f; no voters:
I20260620 14:19:11.662891 2702 raft_consensus.cc:2781] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:11.662941 2702 raft_consensus.cc:482] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:11.662983 2702 raft_consensus.cc:3037] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:11.752640 2702 raft_consensus.cc:504] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.752862 2702 leader_election.cc:288] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 election: Requested vote from peers 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713), d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763)
I20260620 14:19:11.753144 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da28e276fe04f67ae87aeb4c42af950" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2f224c84884a90a10534795943ca1f"
I20260620 14:19:11.753149 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da28e276fe04f67ae87aeb4c42af950" candidate_uuid: "bece832cb4f04af5ad601ff7a487d69f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e957fae7d74418a19bde5cbbfc2716"
I20260620 14:19:11.753264 2418 raft_consensus.cc:3037] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:11.753280 2278 raft_consensus.cc:3037] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:11.754024 2278 raft_consensus.cc:2454] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 1.
I20260620 14:19:11.754172 2418 raft_consensus.cc:2454] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bece832cb4f04af5ad601ff7a487d69f in term 1.
I20260620 14:19:11.754249 2490 leader_election.cc:302] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2f224c84884a90a10534795943ca1f, bece832cb4f04af5ad601ff7a487d69f; no voters:
I20260620 14:19:11.754360 2702 raft_consensus.cc:2781] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:11.754442 2702 raft_consensus.cc:686] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 1 LEADER]: Becoming Leader. State: Replica: bece832cb4f04af5ad601ff7a487d69f, State: Running, Role: LEADER
I20260620 14:19:11.754499 2702 consensus_queue.cc:237] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } }
I20260620 14:19:11.755270 2120 catalog_manager.cc:5495] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f reported cstate change: term changed from 0 to 1, leader changed from <none> to bece832cb4f04af5ad601ff7a487d69f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bece832cb4f04af5ad601ff7a487d69f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bece832cb4f04af5ad601ff7a487d69f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35009 } health_report { overall_health: HEALTHY } } }
I20260620 14:19:11.817754 2418 raft_consensus.cc:1261] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Refusing update from remote peer bece832cb4f04af5ad601ff7a487d69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:11.817785 2278 raft_consensus.cc:1261] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Refusing update from remote peer bece832cb4f04af5ad601ff7a487d69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:11.818071 2705 consensus_queue.cc:1035] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5e957fae7d74418a19bde5cbbfc2716" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:11.818205 2702 consensus_queue.cc:1035] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2f224c84884a90a10534795943ca1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:19:11.822647 2693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:19:13.342980 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:13.343096 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:13.343910 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55e1d9219458429293212124bf05cde0: Sending DeleteTablet for 3 replicas of tablet 9da28e276fe04f67ae87aeb4c42af950
I20260620 14:19:13.344154 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da28e276fe04f67ae87aeb4c42af950 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37648
I20260620 14:19:13.344182 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da28e276fe04f67ae87aeb4c42af950 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260620 14:19:13.344200 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da28e276fe04f67ae87aeb4c42af950 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55014
I20260620 14:19:13.344355 2763 tablet_replica.cc:330] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f: stopping tablet replica
I20260620 14:19:13.344422 2763 raft_consensus.cc:2229] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:13.344457 2763 raft_consensus.cc:2258] T 9da28e276fe04f67ae87aeb4c42af950 P 0c2f224c84884a90a10534795943ca1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:13.344481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 2193
I20260620 14:19:13.347687 2765 tablet_replica.cc:330] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: stopping tablet replica
I20260620 14:19:13.347767 2765 raft_consensus.cc:2229] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:13.347807 2765 raft_consensus.cc:2258] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:13.348104 2765 ts_tablet_manager.cc:1905] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:13.347903 2764 tablet_replica.cc:330] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: stopping tablet replica
I20260620 14:19:13.350095 2764 raft_consensus.cc:2229] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:13.350172 2765 ts_tablet_manager.cc:1918] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:19:13.350216 2764 raft_consensus.cc:2258] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:13.350239 2765 log.cc:1196] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/9da28e276fe04f67ae87aeb4c42af950
I20260620 14:19:13.350454 2764 ts_tablet_manager.cc:1905] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:13.350482 2765 ts_tablet_manager.cc:1939] T 9da28e276fe04f67ae87aeb4c42af950 P d5e957fae7d74418a19bde5cbbfc2716: Deleting consensus metadata
I20260620 14:19:13.350874 2106 catalog_manager.cc:4841] TS d5e957fae7d74418a19bde5cbbfc2716 (127.0.98.2:39763): tablet 9da28e276fe04f67ae87aeb4c42af950 (table test-table [id=f6255f58f7d7417b98a500d8727bb94c]) successfully deleted
W20260620 14:19:13.352358 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36713 (error 108)
I20260620 14:19:13.352456 2764 ts_tablet_manager.cc:1918] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:19:13.352506 2764 log.cc:1196] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/9da28e276fe04f67ae87aeb4c42af950
I20260620 14:19:13.352785 2764 ts_tablet_manager.cc:1939] T 9da28e276fe04f67ae87aeb4c42af950 P bece832cb4f04af5ad601ff7a487d69f: Deleting consensus metadata
I20260620 14:19:13.352982 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:13.352769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36713 (local address 127.0.0.1:37668)
0620 14:19:13.352890 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:13.352938 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260620 14:19:13.353111 2107 catalog_manager.cc:4841] TS bece832cb4f04af5ad601ff7a487d69f (127.0.98.3:35009): tablet 9da28e276fe04f67ae87aeb4c42af950 (table test-table [id=f6255f58f7d7417b98a500d8727bb94c]) successfully deleted
W20260620 14:19:13.353125 2106 catalog_manager.cc:4584] TS 0c2f224c84884a90a10534795943ca1f (127.0.98.1:36713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9da28e276fe04f67ae87aeb4c42af950: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36713: connect: Connection refused (error 111)
I20260620 14:19:13.353314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 2333
I20260620 14:19:13.359433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 2473
I20260620 14:19:13.364895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 2089
2026-06-20T14:19:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5754 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:13 dist-test-slave-h6n5 krb5kdc[2770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:13 dist-test-slave-h6n5 krb5kdc[2770](info): set up 2 sockets
Jun 20 14:19:13 dist-test-slave-h6n5 krb5kdc[2770](info): commencing operation
krb5kdc: starting...
W20260620 14:19:14.403414 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:14 dist-test-slave-h6n5 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:14.519796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:14.597903 2783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:14.598160 2783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:14.598198 2783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:14.599378 2783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:14.599452 2783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:14.599474 2783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:14.599493 2783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:14.599525 2783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:14.601078 2783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:14.601363 2783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:14.601599 2783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:14.607003 2791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:14.607007 2788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:14.607033 2783 server_base.cc:1034] running on GCE node
W20260620 14:19:14.607057 2789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:14.607523 2783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:14.607882 2783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:14.609084 2783 hybrid_clock.cc:648] HybridClock initialized: now 1781965154609070 us; error 27 us; skew 500 ppm
Jun 20 14:19:14 dist-test-slave-h6n5 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:14.611737 2783 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:14.612181 2783 webserver.cc:460] Webserver started at http://127.0.98.62:35823/ using document root <none> and password file <none>
I20260620 14:19:14.612428 2783 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:14.612499 2783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:14.612650 2783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:14.613794 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "492f2db669f6d3de527179cdc5f05c21"
server_key_iv: "78ed6d51fd68f4589ec4ee196bcc35ad"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.614148 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "492f2db669f6d3de527179cdc5f05c21"
server_key_iv: "78ed6d51fd68f4589ec4ee196bcc35ad"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.615682 2783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:14.616616 2798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:14.616892 2783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:14.617015 2783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "492f2db669f6d3de527179cdc5f05c21"
server_key_iv: "78ed6d51fd68f4589ec4ee196bcc35ad"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.617095 2783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:14.628674 2783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:14.630910 2783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:14.631099 2783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:14.635054 2783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42517
I20260620 14:19:14.635102 2850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42517 every 8 connection(s)
I20260620 14:19:14.635717 2783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:14.636411 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:14.638788 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a: Bootstrap starting.
I20260620 14:19:14.639423 2851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:14.639763 2851 log.cc:824] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:14.640565 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a: No bootstrap required, opened a new log
I20260620 14:19:14.642200 2851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260620 14:19:14.642346 2851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:14.642369 2851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676ba7a1eb1f49ec9f5da7bdd8733e9a, State: Initialized, Role: FOLLOWER
I20260620 14:19:14.642546 2851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260620 14:19:14.642608 2851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:14.642629 2851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:14.642688 2851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:14.643693 2851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260620 14:19:14.643805 2851 leader_election.cc:302] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 676ba7a1eb1f49ec9f5da7bdd8733e9a; no voters:
I20260620 14:19:14.644011 2851 leader_election.cc:288] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:14.644047 2854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:14.644029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 2783
I20260620 14:19:14.644137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:14.644186 2854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [term 1 LEADER]: Becoming Leader. State: Replica: 676ba7a1eb1f49ec9f5da7bdd8733e9a, State: Running, Role: LEADER
I20260620 14:19:14.644259 2854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260620 14:19:14.644289 2851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:14.644352 392 external_mini_cluster.cc:1428] Setting key 6305079c43dcf9f4785b53e7efda760b
I20260620 14:19:14.644562 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 676ba7a1eb1f49ec9f5da7bdd8733e9a. Latest consensus state: current_term: 1 leader_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } } }
I20260620 14:19:14.644747 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:14.644600 2855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676ba7a1eb1f49ec9f5da7bdd8733e9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } } }
I20260620 14:19:14.645213 2855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:14.645402 2865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:14.646140 2865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:14.647639 2865 catalog_manager.cc:1347] Generated new cluster ID: cf826bb8ab17481d95f47beb408cb586
I20260620 14:19:14.647702 2865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:14 dist-test-slave-h6n5 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:14.656364 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:14.644878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.0.98.62:42517)
0620 14:19:14.645073 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:14.645083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:14.645125 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:19:14.645264 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:14.645272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:14.645329 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:14.646322 (+ 993us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:14.646416 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:14.647800 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:14.648846 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:14.649517 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:14.649633 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:14.652434 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:14.652438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:14.652438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:14.652647 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:14.654668 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:14.655080 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:14.655083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:14.655083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:14.655190 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:14.655404 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:14.655405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:14.655405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:14.655729 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:14.655868 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:14.655998 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:19:14.656132 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260620 14:19:14.668102 2865 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:14.668731 2865 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:14.673122 2865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 676ba7a1eb1f49ec9f5da7bdd8733e9a: Generated new TSK 0
I20260620 14:19:14.673339 2865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:14.707397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42517
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:14.814885 2877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:14.815089 2877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:14.815120 2877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:14.816352 2877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:14.816395 2877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:14.816422 2877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:14.817926 2877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:14.818174 2877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:14.818405 2877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:14.823406 2883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:14.823446 2885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:14.823498 2882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:14.823765 2877 server_base.cc:1034] running on GCE node
I20260620 14:19:14.823944 2877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:14.824131 2877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:14.825270 2877 hybrid_clock.cc:648] HybridClock initialized: now 1781965154825254 us; error 32 us; skew 500 ppm
Jun 20 14:19:14 dist-test-slave-h6n5 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:14.828007 2877 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:14.828485 2877 webserver.cc:460] Webserver started at http://127.0.98.1:38553/ using document root <none> and password file <none>
I20260620 14:19:14.828719 2877 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:14.828765 2877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:14.828900 2877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:14.829958 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "f98ee482a6bc49a1a063b8e050195ec4"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "468f29de62c8d9d55be6bde4ffa64103"
server_key_iv: "038e5e3f8534079aa1541118e75038b5"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.830315 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f98ee482a6bc49a1a063b8e050195ec4"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "468f29de62c8d9d55be6bde4ffa64103"
server_key_iv: "038e5e3f8534079aa1541118e75038b5"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.831673 2877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:14.832473 2892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:14.832693 2877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:14.832804 2877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "f98ee482a6bc49a1a063b8e050195ec4"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "468f29de62c8d9d55be6bde4ffa64103"
server_key_iv: "038e5e3f8534079aa1541118e75038b5"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.832877 2877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:14.853820 2877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:14.855877 2877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:14.856035 2877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:14.856308 2877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:14.856633 2877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:14.856689 2877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:14.856724 2877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:14.856755 2877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:14.862566 2877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33657
I20260620 14:19:14.862610 3005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33657 every 8 connection(s)
I20260620 14:19:14.862928 2877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:14.863219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 2877
I20260620 14:19:14.863339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:14.863509 392 external_mini_cluster.cc:1428] Setting key 6ca503f448e2f3ff71cc97ced58c6b29
Jun 20 14:19:14 dist-test-slave-h6n5 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:19:14.874789 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:14.863712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51163 (local address 127.0.98.62:42517)
0620 14:19:14.863767 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:14.863770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:14.864070 (+ 300us) server_negotiation.cc:407] Connection header received
0620 14:19:14.865432 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:14.865434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:14.865448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:14.865503 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:14.866300 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:14.866806 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:14.868439 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:14.868700 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:14.868771 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:14.871407 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:14.871409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:14.871409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:14.871420 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:14.873149 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:14.873662 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:14.873664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:14.873664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:14.873754 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:14.874048 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:14.874049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:14.874049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:14.874181 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:14.874255 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:14.874589 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 14:19:14.874684 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:19:14.875099 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:14.863746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42517 (local address 127.0.98.1:51163)
0620 14:19:14.863940 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:14.863986 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:14.865303 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:14.865612 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:14.865616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:14.865986 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:14.866183 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:14.866189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:14.866917 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:14.866918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:14.868328 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:14.868331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:14.868801 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:14.868802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:14.868959 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:14.869554 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:14.869566 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:14.871295 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:14.873260 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:14.873270 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:14.873273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:14.873557 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:14.873838 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:14.873839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:14.873839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:14.873966 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:14.874257 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:14.874265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:14.874417 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 14:19:14.874589 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:19:14.874775 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:19:14.875666 3006 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42517
I20260620 14:19:14.875763 3006 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:14.876003 3006 heartbeater.cc:511] Master 127.0.98.62:42517 requested a full tablet report, sending...
I20260620 14:19:14.876724 2815 ts_manager.cc:194] Registered new tserver with Master: f98ee482a6bc49a1a063b8e050195ec4 (127.0.98.1:33657)
I20260620 14:19:14.877445 2815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:14.891636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42517
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:14.972775 3011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:14.973027 3011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:14.973065 3011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:14.974273 3011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:14.974354 3011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:14.974399 3011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:14.976009 3011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:14.976289 3011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:14.976537 3011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:14.981703 3016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:14.981886 3011 server_base.cc:1034] running on GCE node
W20260620 14:19:14.981709 3019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:14.981753 3017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:14.982241 3011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:14.982457 3011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:14.983608 3011 hybrid_clock.cc:648] HybridClock initialized: now 1781965154983592 us; error 44 us; skew 500 ppm
Jun 20 14:19:14 dist-test-slave-h6n5 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:14.986287 3011 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:14.986786 3011 webserver.cc:460] Webserver started at http://127.0.98.2:38743/ using document root <none> and password file <none>
I20260620 14:19:14.987030 3011 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:14.987099 3011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:14.987253 3011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:14.988400 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "b34f371e894544d4897c435883134696"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "837ba14cf9d427dddabf2216342d6546"
server_key_iv: "32cc83d68f33cfac5eb7d6935fb64f34"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.988798 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b34f371e894544d4897c435883134696"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "837ba14cf9d427dddabf2216342d6546"
server_key_iv: "32cc83d68f33cfac5eb7d6935fb64f34"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.990146 3011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:19:14.990960 3026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:14.991138 3011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:14.991232 3011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "b34f371e894544d4897c435883134696"
format_stamp: "Formatted at 2026-06-20 14:19:14 on dist-test-slave-h6n5"
server_key: "837ba14cf9d427dddabf2216342d6546"
server_key_iv: "32cc83d68f33cfac5eb7d6935fb64f34"
server_key_version: "encryptionkey@0"
I20260620 14:19:14.991320 3011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:14.998654 3011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:15.001116 3011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:15.001284 3011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:15.001523 3011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:15.001842 3011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:15.001899 3011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:15.001941 3011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:15.001969 3011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:15.008184 3011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42983
I20260620 14:19:15.008229 3139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42983 every 8 connection(s)
I20260620 14:19:15.008584 3011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:15 dist-test-slave-h6n5 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:15.016345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3011
I20260620 14:19:15.016448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:15.016731 392 external_mini_cluster.cc:1428] Setting key a9518b66d3fe0df7f095083c1e074f6c
I20260620 14:19:15.018363 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:15.009317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42517 (local address 127.0.98.2:32875)
0620 14:19:15.009477 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:15.009518 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:15.010430 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:15.010662 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:15.010665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:15.010922 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:15.011119 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:15.011123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.011855 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.011855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:15.012650 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:15.012652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.013042 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.013042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:15.013137 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:15.013717 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:15.013726 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:15.015061 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:15.016916 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:15.016922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:15.016925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:15.017220 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:15.017454 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:15.017455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:15.017455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:15.017592 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:15.017881 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:15.017885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:15.017998 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 14:19:15.018081 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:19:15.018203 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 14:19:15.018370 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:15.009357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32875 (local address 127.0.98.62:42517)
0620 14:19:15.009425 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:15.009427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:15.009606 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:19:15.010520 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:15.010521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:15.010531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:15.010606 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:15.011224 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.011753 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.012767 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.012967 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.013025 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:15.015164 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:15.015166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:15.015166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:15.015173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:15.016819 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:15.017296 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:15.017297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:15.017297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:15.017349 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:15.017666 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:15.017667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:15.017667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:15.017814 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:15.017884 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:15.018140 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:19:15.018237 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:15.019363 3140 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42517
I20260620 14:19:15.019529 3140 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:15.019761 3140 heartbeater.cc:511] Master 127.0.98.62:42517 requested a full tablet report, sending...
I20260620 14:19:15.020167 2815 ts_manager.cc:194] Registered new tserver with Master: b34f371e894544d4897c435883134696 (127.0.98.2:42983)
I20260620 14:19:15.020618 2815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32875
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:15.042050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42517
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:15.122458 3145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:15.122665 3145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:15.122695 3145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:15.123806 3145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:15.123842 3145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:15.123868 3145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:15.125267 3145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:15.125443 3145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:15.125607 3145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:15.130432 3150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:15.130456 3151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:15.130506 3153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:15.130627 3145 server_base.cc:1034] running on GCE node
I20260620 14:19:15.130964 3145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:15.131225 3145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:15.132393 3145 hybrid_clock.cc:648] HybridClock initialized: now 1781965155132389 us; error 52 us; skew 500 ppm
Jun 20 14:19:15 dist-test-slave-h6n5 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:15.134809 3145 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:15.135329 3145 webserver.cc:460] Webserver started at http://127.0.98.3:41803/ using document root <none> and password file <none>
I20260620 14:19:15.135596 3145 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:15.135656 3145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:15.135788 3145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:15.136904 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "19bc0a79adfe4cfea49bce76204a82dd"
format_stamp: "Formatted at 2026-06-20 14:19:15 on dist-test-slave-h6n5"
server_key: "f7b344bbbe13e2525ac231d15b5e3743"
server_key_iv: "01134af91f453fac09ce17d38636f56a"
server_key_version: "encryptionkey@0"
I20260620 14:19:15.137264 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "19bc0a79adfe4cfea49bce76204a82dd"
format_stamp: "Formatted at 2026-06-20 14:19:15 on dist-test-slave-h6n5"
server_key: "f7b344bbbe13e2525ac231d15b5e3743"
server_key_iv: "01134af91f453fac09ce17d38636f56a"
server_key_version: "encryptionkey@0"
I20260620 14:19:15.138566 3145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:15.139346 3160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:15.139621 3145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:15.139729 3145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "19bc0a79adfe4cfea49bce76204a82dd"
format_stamp: "Formatted at 2026-06-20 14:19:15 on dist-test-slave-h6n5"
server_key: "f7b344bbbe13e2525ac231d15b5e3743"
server_key_iv: "01134af91f453fac09ce17d38636f56a"
server_key_version: "encryptionkey@0"
I20260620 14:19:15.139818 3145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:15.160265 3145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:15.162474 3145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:15.162652 3145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:15.162915 3145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:15.163229 3145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:15.163287 3145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:15.163329 3145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:15.163353 3145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:15.169139 3145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35685
I20260620 14:19:15.169188 3273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35685 every 8 connection(s)
I20260620 14:19:15.169531 3145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:19:15 dist-test-slave-h6n5 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:15.177026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3145
I20260620 14:19:15.177130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:15.177322 392 external_mini_cluster.cc:1428] Setting key dd996e919439c87870e81bfb71741d69
I20260620 14:19:15.179483 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:15.170237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36553 (local address 127.0.98.62:42517)
0620 14:19:15.170282 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:15.170284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:15.170511 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:19:15.171434 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:15.171436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:15.171447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:15.171509 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:15.172184 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.172675 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.173840 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.174037 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.174106 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:15.176386 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:15.176387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:15.176388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:15.176395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:15.178087 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:15.178485 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:15.178486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:15.178487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:15.178562 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:15.178817 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:15.178818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:15.178818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:15.178944 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:15.179021 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:15.179301 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:19:15.179388 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:19:15.179747 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:15.170202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42517 (local address 127.0.98.3:36553)
0620 14:19:15.170374 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:15.170418 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:15.171340 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:15.171631 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:15.171636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:15.171911 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:15.172102 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:15.172106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.172755 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.172756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:15.173759 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:15.173761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:15.174119 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:15.174120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:15.174230 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:15.174774 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:15.174783 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:15.176286 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:15.178151 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:15.178159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:15.178162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:15.178392 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:15.178631 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:15.178632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:15.178632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:15.178747 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:15.179013 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:15.179019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:15.179164 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 14:19:15.179280 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:19:15.179448 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260620 14:19:15.180281 3274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42517
I20260620 14:19:15.180383 3274 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:15.180531 3274 heartbeater.cc:511] Master 127.0.98.62:42517 requested a full tablet report, sending...
I20260620 14:19:15.180892 2815 ts_manager.cc:194] Registered new tserver with Master: 19bc0a79adfe4cfea49bce76204a82dd (127.0.98.3:35685)
I20260620 14:19:15.181237 2815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36553
I20260620 14:19:15.189303 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:19:15.200690 2864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:15.199609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.0.98.62:42517)
0620 14:19:15.199728 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:15.199731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:15.199834 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:19:15.199935 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:15.199937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:15.199956 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:15.200031 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:15.200596 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44594: BlockingRecv error: recv got EOF from 127.0.0.1:44594 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:19:15.201606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 2877
I20260620 14:19:15.206166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3011
I20260620 14:19:15.210621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3145
I20260620 14:19:15.214918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 2783
2026-06-20T14:19:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1849 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:15 dist-test-slave-h6n5 krb5kdc[3286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:15 dist-test-slave-h6n5 krb5kdc[3286](info): set up 2 sockets
Jun 20 14:19:15 dist-test-slave-h6n5 krb5kdc[3286](info): commencing operation
krb5kdc: starting...
W20260620 14:19:16.247478 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:16 dist-test-slave-h6n5 krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:16.362634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:16.442023 3299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:16.442288 3299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:16.442322 3299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:16.443473 3299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:16.443548 3299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:16.443600 3299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:16.443624 3299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:16.443652 3299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:16.445488 3299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:16.445801 3299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:16.446043 3299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:16.450749 3304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:16.450765 3307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:16.450765 3305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:16.450958 3299 server_base.cc:1034] running on GCE node
I20260620 14:19:16.451256 3299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:16.451612 3299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:16.452766 3299 hybrid_clock.cc:648] HybridClock initialized: now 1781965156452743 us; error 39 us; skew 500 ppm
Jun 20 14:19:16 dist-test-slave-h6n5 krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:16.455338 3299 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:16.455809 3299 webserver.cc:460] Webserver started at http://127.0.98.62:39143/ using document root <none> and password file <none>
I20260620 14:19:16.455991 3299 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:16.456038 3299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:16.456139 3299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:16.457226 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7"
format_stamp: "Formatted at 2026-06-20 14:19:16 on dist-test-slave-h6n5"
server_key: "31dbbe2a627aa786ffa09b8ba718a369"
server_key_iv: "866917013106afd9df31668bb5272ef4"
server_key_version: "encryptionkey@0"
I20260620 14:19:16.457533 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7"
format_stamp: "Formatted at 2026-06-20 14:19:16 on dist-test-slave-h6n5"
server_key: "31dbbe2a627aa786ffa09b8ba718a369"
server_key_iv: "866917013106afd9df31668bb5272ef4"
server_key_version: "encryptionkey@0"
I20260620 14:19:16.458863 3299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:16.459754 3314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:16.459944 3299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:16.460032 3299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7"
format_stamp: "Formatted at 2026-06-20 14:19:16 on dist-test-slave-h6n5"
server_key: "31dbbe2a627aa786ffa09b8ba718a369"
server_key_iv: "866917013106afd9df31668bb5272ef4"
server_key_version: "encryptionkey@0"
I20260620 14:19:16.460088 3299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:16.473184 3299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:16.475490 3299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:16.475669 3299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:16.479317 3299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43749
I20260620 14:19:16.479405 3366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43749 every 8 connection(s)
I20260620 14:19:16.479771 3299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:16.480672 3367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:16.482939 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7: Bootstrap starting.
I20260620 14:19:16.483522 3367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:16.483886 3367 log.cc:824] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:16.484665 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7: No bootstrap required, opened a new log
I20260620 14:19:16.486217 3367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43749 } }
I20260620 14:19:16.486384 3367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:16.486433 3367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7703ed29d3eb40a3ac71d9de14b0d1b7, State: Initialized, Role: FOLLOWER
I20260620 14:19:16.486604 3367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43749 } }
I20260620 14:19:16.486723 3367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:16.486769 3367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:16.486817 3367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:16.486913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3299
I20260620 14:19:16.486994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:16.487187 392 external_mini_cluster.cc:1428] Setting key 1bf1940048508dacd58ab1a18d328943
I20260620 14:19:16.487869 3367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43749 } }
I20260620 14:19:16.487998 3367 leader_election.cc:302] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7703ed29d3eb40a3ac71d9de14b0d1b7; no voters:
I20260620 14:19:16.488162 3367 leader_election.cc:288] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:16.488204 3372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:16.488308 3372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7703ed29d3eb40a3ac71d9de14b0d1b7, State: Running, Role: LEADER
I20260620 14:19:16.488432 3367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:16.488448 3372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43749 } }
I20260620 14:19:16.488794 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43749 } } }
I20260620 14:19:16.488962 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:16.489151 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7703ed29d3eb40a3ac71d9de14b0d1b7. Latest consensus state: current_term: 1 leader_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7703ed29d3eb40a3ac71d9de14b0d1b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43749 } } }
I20260620 14:19:16.489259 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:16.489557 3381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:16.490162 3381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:16.491814 3381 catalog_manager.cc:1347] Generated new cluster ID: 9fe68a416425445d954be3bec8da1e5a
I20260620 14:19:16.491863 3381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:16 dist-test-slave-h6n5 krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:16.498153 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:16.487693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.0.98.62:43749)
0620 14:19:16.487892 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:16.487901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:16.487935 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:19:16.488076 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:16.488083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:16.488140 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:16.488298 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:16.489157 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:16.490477 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:16.491377 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:16.491957 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:16.492089 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:16.494370 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:16.494374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:16.494374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:16.494385 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:16.496267 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:16.496680 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:16.496683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:16.496684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:16.497085 (+ 401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:16.497315 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:16.497317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:16.497317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:16.497557 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:16.497738 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:16.497807 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:19:16.497952 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260620 14:19:16.499127 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:19:16.514919 3381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:16.515491 3381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:16.516987 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:16.512725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.98.62:43749)
0620 14:19:16.512766 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:16.512768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:16.512931 (+ 163us) server_negotiation.cc:407] Connection header received
0620 14:19:16.512973 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:16.512974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:16.512987 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:16.513053 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:16.513448 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:16.514211 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:16.515168 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:16.515495 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:16.515620 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:16.516471 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:16.516472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:16.516473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:16.516717 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:16.516737 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:16.516744 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:16.516808 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:16.516865 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:19:16.516908 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:19:16.522297 3381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7703ed29d3eb40a3ac71d9de14b0d1b7: Generated new TSK 0
I20260620 14:19:16.522492 3381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:16.527844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3299
2026-06-20T14:19:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1310 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:16 dist-test-slave-h6n5 krb5kdc[3400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:16 dist-test-slave-h6n5 krb5kdc[3400](info): set up 2 sockets
Jun 20 14:19:16 dist-test-slave-h6n5 krb5kdc[3400](info): commencing operation
krb5kdc: starting...
W20260620 14:19:17.575395 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:17 dist-test-slave-h6n5 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:17.695125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:17.776845 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:17.777076 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:17.777100 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:17.778230 3413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:17.778267 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:17.778281 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:17.778294 3413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:17.778306 3413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:17.780169 3413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:17.780391 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:17.780587 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:17.785480 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:17.785668 3413 server_base.cc:1034] running on GCE node
W20260620 14:19:17.785492 3421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:17.785491 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:17.786049 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:17.786324 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:17.787501 3413 hybrid_clock.cc:648] HybridClock initialized: now 1781965157787471 us; error 47 us; skew 500 ppm
Jun 20 14:19:17 dist-test-slave-h6n5 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:17.790177 3413 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:17.790654 3413 webserver.cc:460] Webserver started at http://127.0.98.62:41233/ using document root <none> and password file <none>
I20260620 14:19:17.790910 3413 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:17.790956 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:17.791101 3413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:17.792287 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "e26f8cb0f62f4a1fac69feedf84fcc04"
format_stamp: "Formatted at 2026-06-20 14:19:17 on dist-test-slave-h6n5"
server_key: "54eac0a8432610908c7b629b5b69b0e3"
server_key_iv: "5661c20614543501bfb501ab746f6474"
server_key_version: "encryptionkey@0"
I20260620 14:19:17.792683 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "e26f8cb0f62f4a1fac69feedf84fcc04"
format_stamp: "Formatted at 2026-06-20 14:19:17 on dist-test-slave-h6n5"
server_key: "54eac0a8432610908c7b629b5b69b0e3"
server_key_iv: "5661c20614543501bfb501ab746f6474"
server_key_version: "encryptionkey@0"
I20260620 14:19:17.794059 3413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:17.794859 3428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:17.795040 3413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:17.795148 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "e26f8cb0f62f4a1fac69feedf84fcc04"
format_stamp: "Formatted at 2026-06-20 14:19:17 on dist-test-slave-h6n5"
server_key: "54eac0a8432610908c7b629b5b69b0e3"
server_key_iv: "5661c20614543501bfb501ab746f6474"
server_key_version: "encryptionkey@0"
I20260620 14:19:17.795230 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:17.805780 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:17.807770 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:17.808034 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:17.812189 3413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41187
I20260620 14:19:17.812235 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41187 every 8 connection(s)
I20260620 14:19:17.812582 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:17.813534 3481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:17.815908 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04: Bootstrap starting.
I20260620 14:19:17.816493 3481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:17.816804 3481 log.cc:824] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:17.817521 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04: No bootstrap required, opened a new log
I20260620 14:19:17.819164 3481 raft_consensus.cc:348] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260620 14:19:17.819502 3481 raft_consensus.cc:374] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:17.819543 3481 raft_consensus.cc:729] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26f8cb0f62f4a1fac69feedf84fcc04, State: Initialized, Role: FOLLOWER
I20260620 14:19:17.819826 3481 consensus_queue.cc:260] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260620 14:19:17.819926 3481 raft_consensus.cc:388] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:17.819952 3481 raft_consensus.cc:482] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:17.820015 3481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:17.820005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3413
I20260620 14:19:17.820122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:17.820312 392 external_mini_cluster.cc:1428] Setting key 7ec0ea82690c3abaa65148b171439ac9
I20260620 14:19:17.821269 3481 raft_consensus.cc:504] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260620 14:19:17.821380 3481 leader_election.cc:302] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e26f8cb0f62f4a1fac69feedf84fcc04; no voters:
I20260620 14:19:17.821592 3481 leader_election.cc:288] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:17.821638 3486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:17.821832 3486 raft_consensus.cc:686] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [term 1 LEADER]: Becoming Leader. State: Replica: e26f8cb0f62f4a1fac69feedf84fcc04, State: Running, Role: LEADER
I20260620 14:19:17.821858 3481 sys_catalog.cc:564] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:17.821933 3486 consensus_queue.cc:237] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } }
I20260620 14:19:17.822368 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e26f8cb0f62f4a1fac69feedf84fcc04. Latest consensus state: current_term: 1 leader_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } } }
I20260620 14:19:17.822376 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26f8cb0f62f4a1fac69feedf84fcc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41187 } } }
I20260620 14:19:17.822522 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:17.822710 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:17.822746 3495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:17.823356 3495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:17.825242 3495 catalog_manager.cc:1347] Generated new cluster ID: 11befdff34a6437691ea03428e4484c3
I20260620 14:19:17.825294 3495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:17 dist-test-slave-h6n5 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:17.832655 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:17.820834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.0.98.62:41187)
0620 14:19:17.821081 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:17.821092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:17.821130 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:19:17.821252 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:17.821260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:17.821321 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:17.821484 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:17.822370 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:17.824013 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:17.825071 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:17.825670 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:17.825779 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:17.828786 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:17.828790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:17.828791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:17.828803 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:17.830855 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:17.831347 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:17.831351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:17.831351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:17.831475 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:17.831723 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:17.831724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:17.831724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:17.832034 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:17.832182 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:17.832314 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:19:17.832448 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":120,"threads_started":1}
I20260620 14:19:17.835171 3495 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:17.835774 3495 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:17.839296 3495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e26f8cb0f62f4a1fac69feedf84fcc04: Generated new TSK 0
I20260620 14:19:17.839476 3495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:17.871337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41187
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:17.956687 3507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:17.956930 3507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:17.956957 3507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:17.958249 3507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:17.958300 3507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:17.958331 3507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:17.959968 3507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:17.960233 3507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:17.960490 3507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:17.965696 3512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:17.965770 3515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:17.965771 3513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:17.966145 3507 server_base.cc:1034] running on GCE node
I20260620 14:19:17.966362 3507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:17.966622 3507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:17.967818 3507 hybrid_clock.cc:648] HybridClock initialized: now 1781965157967818 us; error 48 us; skew 500 ppm
Jun 20 14:19:17 dist-test-slave-h6n5 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:17.970422 3507 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:17.970945 3507 webserver.cc:460] Webserver started at http://127.0.98.1:35899/ using document root <none> and password file <none>
I20260620 14:19:17.971189 3507 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:17.971264 3507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:17.971424 3507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:17.972645 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "39ecb2a3d7e94580adff728365aeef28"
format_stamp: "Formatted at 2026-06-20 14:19:17 on dist-test-slave-h6n5"
server_key: "58d31d080af661a7891a8cd868b0e9ce"
server_key_iv: "ac2562b6843fc35da401c1feb4cc825f"
server_key_version: "encryptionkey@0"
I20260620 14:19:17.973058 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "39ecb2a3d7e94580adff728365aeef28"
format_stamp: "Formatted at 2026-06-20 14:19:17 on dist-test-slave-h6n5"
server_key: "58d31d080af661a7891a8cd868b0e9ce"
server_key_iv: "ac2562b6843fc35da401c1feb4cc825f"
server_key_version: "encryptionkey@0"
I20260620 14:19:17.974496 3507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:17.975387 3522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:17.975648 3507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:17.975805 3507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "39ecb2a3d7e94580adff728365aeef28"
format_stamp: "Formatted at 2026-06-20 14:19:17 on dist-test-slave-h6n5"
server_key: "58d31d080af661a7891a8cd868b0e9ce"
server_key_iv: "ac2562b6843fc35da401c1feb4cc825f"
server_key_version: "encryptionkey@0"
I20260620 14:19:17.975893 3507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:17.988337 3507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:17.990541 3507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:17.990728 3507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:17.990986 3507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:17.991318 3507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:17.991375 3507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:17.991420 3507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:17.991447 3507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:17.997642 3507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45789
I20260620 14:19:17.997689 3635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45789 every 8 connection(s)
I20260620 14:19:17.998172 3507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:18.006441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3507
I20260620 14:19:18.006552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:18.006775 392 external_mini_cluster.cc:1428] Setting key 72f9372220dc4b8da330a6f2429ac3e4
I20260620 14:19:18.008107 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:17.999016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41187 (local address 127.0.98.1:55023)
0620 14:19:17.999192 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:17.999231 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:17.999806 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:18.000070 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:18.000074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:18.000673 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:18.000890 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:18.000895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.001648 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.001649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:18.002486 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:18.002488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.002848 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.002849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:18.002953 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:18.003453 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:18.003461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:18.004980 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:18.006437 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:18.006445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:18.006448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:18.006755 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:18.007074 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:18.007074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:18.007075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:18.007175 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:18.007520 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:18.007527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:18.007718 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 14:19:18.007816 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:19:18.007942 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260620 14:19:18.008111 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:17.999027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55023 (local address 127.0.98.62:41187)
0620 14:19:17.999080 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:17.999083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:17.999321 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:19:17.999936 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:17.999938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:17.999949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:18.000010 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:18.001019 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.001539 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.002571 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.002768 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.002830 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:18.005069 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:18.005071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:18.005071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:18.005078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:18.006356 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.006828 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.006829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.006829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.006923 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.007259 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.007260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.007261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.007394 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:18.007496 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:18.007867 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:19:18.007966 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:19:18.008919 3636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41187
I20260620 14:19:18.009035 3636 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:18.009231 3636 heartbeater.cc:511] Master 127.0.98.62:41187 requested a full tablet report, sending...
I20260620 14:19:18.009831 3445 ts_manager.cc:194] Registered new tserver with Master: 39ecb2a3d7e94580adff728365aeef28 (127.0.98.1:45789)
I20260620 14:19:18.010458 3445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55023
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:18.032555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41187
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:18.113440 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:18.113658 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:18.113683 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:18.114885 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:18.114929 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:18.114957 3641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:18.116438 3641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:18.116693 3641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:18.116945 3641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:18.122099 3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:18.122066 3646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:18.122078 3647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:18.122392 3641 server_base.cc:1034] running on GCE node
I20260620 14:19:18.122624 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:18.122882 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:18.124049 3641 hybrid_clock.cc:648] HybridClock initialized: now 1781965158124027 us; error 39 us; skew 500 ppm
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:18.126416 3641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:18.126900 3641 webserver.cc:460] Webserver started at http://127.0.98.2:39009/ using document root <none> and password file <none>
I20260620 14:19:18.127144 3641 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:18.127215 3641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:18.127372 3641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:18.128496 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c6049c939534597b6980a63fbef3dbd"
format_stamp: "Formatted at 2026-06-20 14:19:18 on dist-test-slave-h6n5"
server_key: "bbe3df1f0cb5161ae99c3f38d4cea8f6"
server_key_iv: "a7712586398709bfe1483410daf52fa7"
server_key_version: "encryptionkey@0"
I20260620 14:19:18.128885 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c6049c939534597b6980a63fbef3dbd"
format_stamp: "Formatted at 2026-06-20 14:19:18 on dist-test-slave-h6n5"
server_key: "bbe3df1f0cb5161ae99c3f38d4cea8f6"
server_key_iv: "a7712586398709bfe1483410daf52fa7"
server_key_version: "encryptionkey@0"
I20260620 14:19:18.130234 3641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:18.130975 3656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:18.131140 3641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:18.131239 3641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "9c6049c939534597b6980a63fbef3dbd"
format_stamp: "Formatted at 2026-06-20 14:19:18 on dist-test-slave-h6n5"
server_key: "bbe3df1f0cb5161ae99c3f38d4cea8f6"
server_key_iv: "a7712586398709bfe1483410daf52fa7"
server_key_version: "encryptionkey@0"
I20260620 14:19:18.131318 3641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:18.149291 3641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:18.151616 3641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:18.151783 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:18.152051 3641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:18.152379 3641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:18.152433 3641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:18.152474 3641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:18.152503 3641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:18.158339 3641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37873
I20260620 14:19:18.158409 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37873 every 8 connection(s)
I20260620 14:19:18.158761 3641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:18.167632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3641
I20260620 14:19:18.167755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:18.167930 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:18.159499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39031 (local address 127.0.98.62:41187)
0620 14:19:18.159585 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:18.159587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:18.159776 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:19:18.160328 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:18.160329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:18.160343 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:18.160408 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:18.161384 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.161895 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.162907 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.163116 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.163193 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:18.165290 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:18.165292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:18.165292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:18.165301 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:18.166491 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.166903 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.166905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.166905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.166965 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.167219 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.167220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.167220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.167339 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:18.167410 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:18.167741 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:19:18.167829 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:19:18.167945 392 external_mini_cluster.cc:1428] Setting key 91c9f535269f3c30c3b61512fee482dc
I20260620 14:19:18.168156 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:18.159459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41187 (local address 127.0.98.2:39031)
0620 14:19:18.159642 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:18.159691 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:18.160216 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:18.160480 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:18.160484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:18.161058 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:18.161276 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:18.161282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.161997 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.161999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:18.162805 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:18.162807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.163210 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.163211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:18.163345 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:18.163902 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:18.163909 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:18.165192 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:18.166590 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:18.166597 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:18.166599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:18.166808 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:18.167031 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:18.167032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:18.167033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:18.167142 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:18.167413 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:18.167419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:18.167592 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:19:18.167720 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:19:18.167906 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":117,"threads_started":1}
I20260620 14:19:18.168780 3770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41187
I20260620 14:19:18.168870 3770 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:18.169061 3770 heartbeater.cc:511] Master 127.0.98.62:41187 requested a full tablet report, sending...
I20260620 14:19:18.169417 3445 ts_manager.cc:194] Registered new tserver with Master: 9c6049c939534597b6980a63fbef3dbd (127.0.98.2:37873)
I20260620 14:19:18.169793 3445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39031
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:18.192473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41187
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:18.272616 3775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:18.272828 3775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:18.272853 3775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:18.274029 3775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:18.274075 3775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:18.274101 3775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:18.275519 3775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:18.275797 3775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:18.276031 3775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:18.281154 3781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:18.281136 3780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:18.281159 3783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:18.281383 3775 server_base.cc:1034] running on GCE node
I20260620 14:19:18.281589 3775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:18.281845 3775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:18.282992 3775 hybrid_clock.cc:648] HybridClock initialized: now 1781965158282971 us; error 36 us; skew 500 ppm
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:18.285629 3775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:18.286146 3775 webserver.cc:460] Webserver started at http://127.0.98.3:44907/ using document root <none> and password file <none>
I20260620 14:19:18.286386 3775 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:18.286458 3775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:18.286612 3775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:18.287748 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8f53b2d778d4b8e84c67bfe9e0d1d55"
format_stamp: "Formatted at 2026-06-20 14:19:18 on dist-test-slave-h6n5"
server_key: "dea7ef64d94563f3c0c76a61c08ed43e"
server_key_iv: "05cc64b8a75ecd04173f1ff981df1e8e"
server_key_version: "encryptionkey@0"
I20260620 14:19:18.288105 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f53b2d778d4b8e84c67bfe9e0d1d55"
format_stamp: "Formatted at 2026-06-20 14:19:18 on dist-test-slave-h6n5"
server_key: "dea7ef64d94563f3c0c76a61c08ed43e"
server_key_iv: "05cc64b8a75ecd04173f1ff981df1e8e"
server_key_version: "encryptionkey@0"
I20260620 14:19:18.289521 3775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:18.290241 3790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:18.290436 3775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:18.290547 3775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "d8f53b2d778d4b8e84c67bfe9e0d1d55"
format_stamp: "Formatted at 2026-06-20 14:19:18 on dist-test-slave-h6n5"
server_key: "dea7ef64d94563f3c0c76a61c08ed43e"
server_key_iv: "05cc64b8a75ecd04173f1ff981df1e8e"
server_key_version: "encryptionkey@0"
I20260620 14:19:18.290630 3775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:18.314003 3775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:18.316074 3775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:18.316236 3775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:18.316475 3775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:18.316792 3775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:18.316852 3775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:18.316898 3775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:18.316931 3775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:18.323050 3775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39765
I20260620 14:19:18.323149 3903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39765 every 8 connection(s)
I20260620 14:19:18.323483 3775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:18.327426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3775
I20260620 14:19:18.327538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:18.327762 392 external_mini_cluster.cc:1428] Setting key f48dc54ef36f49d9eaed404beaa4fe14
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:18.332875 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:18.324253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52349 (local address 127.0.98.62:41187)
0620 14:19:18.324295 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:18.324297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:18.324518 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:19:18.325074 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:18.325076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:18.325086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:18.325155 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:18.326136 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.326622 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.327930 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.328136 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.328225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:18.330280 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:18.330282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:18.330282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:18.330291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:18.331465 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.331909 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.331910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.331910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.331971 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.332219 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.332220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.332220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.332337 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:18.332408 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:18.332701 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:19:18.332786 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:19:18.333091 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:18.324212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41187 (local address 127.0.98.3:52349)
0620 14:19:18.324374 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:18.324419 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:18.324936 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:18.325240 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:18.325244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:18.325828 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:18.326037 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:18.326042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.326704 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.326704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:18.327840 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:18.327844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.328242 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.328243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:18.328373 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:18.328842 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:18.328850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:18.330177 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:18.331577 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:18.331585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:18.331587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:18.331817 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:18.332039 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:18.332040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:18.332040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:18.332145 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:18.332412 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:18.332418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:18.332552 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:19:18.332679 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:19:18.332830 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260620 14:19:18.333666 3904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41187
I20260620 14:19:18.333756 3904 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:18.333952 3904 heartbeater.cc:511] Master 127.0.98.62:41187 requested a full tablet report, sending...
I20260620 14:19:18.334329 3445 ts_manager.cc:194] Registered new tserver with Master: d8f53b2d778d4b8e84c67bfe9e0d1d55 (127.0.98.3:39765)
I20260620 14:19:18.334753 3445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52349
I20260620 14:19:18.339536 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965158, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965158, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:18.362576 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:18.355282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.0.98.62:41187)
0620 14:19:18.355362 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:18.355364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:18.355461 (+ 97us) server_negotiation.cc:407] Connection header received
0620 14:19:18.355540 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:18.355541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:18.355571 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:18.355625 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:18.356259 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.356764 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.357519 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:18.357726 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:18.357806 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:18.360093 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:18.360095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:18.360095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:18.360126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:18.361326 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.361724 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.361725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.361726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.361788 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:18.361989 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:18.361990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:18.361990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:18.362151 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:18.362266 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:18.362388 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:19:18.362470 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:19:18.362850 3445 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40624
I20260620 14:19:18.363317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3507
I20260620 14:19:18.368665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3641
I20260620 14:19:18.372740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3775
I20260620 14:19:18.376780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3413
2026-06-20T14:19:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1851 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3916](info): set up 2 sockets
Jun 20 14:19:18 dist-test-slave-h6n5 krb5kdc[3916](info): commencing operation
krb5kdc: starting...
W20260620 14:19:19.419476 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:19 dist-test-slave-h6n5 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:19.538462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:19.619757 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:19.620003 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:19.620040 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:19.621201 3929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:19.621270 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:19.621294 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:19.621316 3929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:19.621344 3929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:19.622774 3929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:19.623049 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:19.623273 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:19.628153 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:19.628173 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:19.628243 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:19.628286 3929 server_base.cc:1034] running on GCE node
I20260620 14:19:19.628584 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:19.628863 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:19.630045 3929 hybrid_clock.cc:648] HybridClock initialized: now 1781965159630031 us; error 31 us; skew 500 ppm
Jun 20 14:19:19 dist-test-slave-h6n5 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:19.632781 3929 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:19.633272 3929 webserver.cc:460] Webserver started at http://127.0.98.62:46833/ using document root <none> and password file <none>
I20260620 14:19:19.633518 3929 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:19.633587 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:19.633740 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:19.634824 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "f466d4357153782f96fd5265779bd990"
server_key_iv: "6f73fd9de5b3ab0107f22fe4496929fd"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.635202 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "f466d4357153782f96fd5265779bd990"
server_key_iv: "6f73fd9de5b3ab0107f22fe4496929fd"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.636650 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:19.637511 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:19.637742 3929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:19:19.637841 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "f466d4357153782f96fd5265779bd990"
server_key_iv: "6f73fd9de5b3ab0107f22fe4496929fd"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.637934 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:19.648166 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:19.650485 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:19.650656 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:19.654654 3929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39837
I20260620 14:19:19.654686 3996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39837 every 8 connection(s)
I20260620 14:19:19.655026 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:19.655939 3997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:19.658121 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f: Bootstrap starting.
I20260620 14:19:19.658715 3997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:19.659022 3997 log.cc:824] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:19.659821 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f: No bootstrap required, opened a new log
I20260620 14:19:19.661553 3997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39837 } }
I20260620 14:19:19.661690 3997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:19.661712 3997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5501c82b1d7f4c84b01bbcf9b4ad297f, State: Initialized, Role: FOLLOWER
I20260620 14:19:19.661819 3997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39837 } }
I20260620 14:19:19.661867 3997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:19.661891 3997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:19.661921 3997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:19.662395 3997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39837 } }
I20260620 14:19:19.662473 3997 leader_election.cc:302] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5501c82b1d7f4c84b01bbcf9b4ad297f; no voters:
I20260620 14:19:19.662598 3997 leader_election.cc:288] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:19.662729 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:19.662843 3997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:19.663107 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [term 1 LEADER]: Becoming Leader. State: Replica: 5501c82b1d7f4c84b01bbcf9b4ad297f, State: Running, Role: LEADER
I20260620 14:19:19.663133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 3929
I20260620 14:19:19.663228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:19.663218 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39837 } }
I20260620 14:19:19.663432 392 external_mini_cluster.cc:1428] Setting key de4cfe1f5b795205bcd7784f5db1f3ba
I20260620 14:19:19.663615 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39837 } } }
I20260620 14:19:19.663916 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:19.663843 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5501c82b1d7f4c84b01bbcf9b4ad297f. Latest consensus state: current_term: 1 leader_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5501c82b1d7f4c84b01bbcf9b4ad297f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39837 } } }
I20260620 14:19:19.664196 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:19.664256 4010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:19.664809 4010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:19.666536 4010 catalog_manager.cc:1347] Generated new cluster ID: 15f45f2158d34de1b2930bdc58ac165b
I20260620 14:19:19.666589 4010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:19 dist-test-slave-h6n5 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:19.674036 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:19.664062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.62:39837)
0620 14:19:19.664225 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:19.664234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:19.664411 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:19:19.664555 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:19.664563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:19.664648 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:19.665612 (+ 964us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:19.665729 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:19.666905 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:19.667744 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:19.668158 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:19.668280 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:19.670687 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:19.670691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:19.670691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:19.670704 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:19.672501 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:19.672881 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:19.672885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:19.672885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:19.672978 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:19.673192 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:19.673193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:19.673194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:19.673431 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:19.673610 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:19.673691 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:19:19.673838 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260620 14:19:19.687436 4010 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:19.688040 4010 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:19.692384 4010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5501c82b1d7f4c84b01bbcf9b4ad297f: Generated new TSK 0
I20260620 14:19:19.692575 4010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:19.718451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39837
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:19.798825 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:19.799079 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:19.799115 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:19.800300 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:19.800370 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:19.800416 4023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:19.801913 4023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:19.802178 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:19.802414 4023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:19.807436 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:19.807438 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:19.807438 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:19.807698 4023 server_base.cc:1034] running on GCE node
I20260620 14:19:19.807934 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:19.808198 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:19.809370 4023 hybrid_clock.cc:648] HybridClock initialized: now 1781965159809368 us; error 40 us; skew 500 ppm
Jun 20 14:19:19 dist-test-slave-h6n5 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:19.812057 4023 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:19.812534 4023 webserver.cc:460] Webserver started at http://127.0.98.1:39307/ using document root <none> and password file <none>
I20260620 14:19:19.812757 4023 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:19.812834 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:19.812987 4023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:19.814036 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c312cf52183407b941aefe0005314fe"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "49fe173af824a5501fbf9d637c9eacf7"
server_key_iv: "a07006b770c7f2f98c122e2a2cabf694"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.814399 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c312cf52183407b941aefe0005314fe"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "49fe173af824a5501fbf9d637c9eacf7"
server_key_iv: "a07006b770c7f2f98c122e2a2cabf694"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.815732 4023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:19.816442 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:19.816596 4023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:19.816695 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "7c312cf52183407b941aefe0005314fe"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "49fe173af824a5501fbf9d637c9eacf7"
server_key_iv: "a07006b770c7f2f98c122e2a2cabf694"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.816766 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:19.824642 4023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:19.827000 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:19.827164 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:19.827406 4023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:19.827764 4023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:19.827816 4023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:19.827852 4023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:19.827900 4023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:19.834033 4023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33001
I20260620 14:19:19.834075 4151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33001 every 8 connection(s)
I20260620 14:19:19.834406 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:19:19 dist-test-slave-h6n5 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:19.843472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4023
I20260620 14:19:19.843591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:19.843803 392 external_mini_cluster.cc:1428] Setting key 63d43d10d20e8f7a3595b74956b486dd
I20260620 14:19:19.844389 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:19.835156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33053 (local address 127.0.98.62:39837)
0620 14:19:19.835262 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:19.835265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:19.835429 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:19:19.836505 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:19.836506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:19.836520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:19.836579 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:19.837227 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:19.837736 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:19.838762 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:19.838981 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:19.839069 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:19.841335 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:19.841337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:19.841338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:19.841362 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:19.842884 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:19.843307 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:19.843308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:19.843309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:19.843374 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:19.843688 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:19.843689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:19.843690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:19.843808 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:19.843890 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:19.844208 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:19:19.844292 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:19:19.844568 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:19.835048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39837 (local address 127.0.98.1:33053)
0620 14:19:19.835288 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:19.835337 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:19.836391 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:19.836649 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:19.836653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:19.836928 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:19.837120 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:19.837124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:19.837834 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:19.837836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:19.838648 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:19.838649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:19.839079 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:19.839080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:19.839208 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:19.839785 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:19.839794 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:19.841232 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:19.842990 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:19.842996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:19.842999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:19.843227 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:19.843444 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:19.843445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:19.843445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:19.843622 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:19.843881 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:19.843887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:19.844049 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 14:19:19.844171 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:19:19.844331 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260620 14:19:19.845019 4152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39837
I20260620 14:19:19.845127 4152 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:19.845327 4152 heartbeater.cc:511] Master 127.0.98.62:39837 requested a full tablet report, sending...
I20260620 14:19:19.845834 3961 ts_manager.cc:194] Registered new tserver with Master: 7c312cf52183407b941aefe0005314fe (127.0.98.1:33001)
I20260620 14:19:19.846458 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33053
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:19.868536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39837
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:19.954946 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:19.955150 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:19.955174 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:19.956394 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:19.956435 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:19.956462 4157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:19.957912 4157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:19.958142 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:19.958386 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:19.963336 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:19.963294 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:19.963295 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:19.963651 4157 server_base.cc:1034] running on GCE node
I20260620 14:19:19.963848 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:19.964036 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:19.965179 4157 hybrid_clock.cc:648] HybridClock initialized: now 1781965159965155 us; error 48 us; skew 500 ppm
Jun 20 14:19:19 dist-test-slave-h6n5 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:19.968370 4157 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:19.968777 4157 webserver.cc:460] Webserver started at http://127.0.98.2:41135/ using document root <none> and password file <none>
I20260620 14:19:19.968961 4157 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:19.969002 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:19.969091 4157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:19.970077 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd7533de6e3841e2933a3a72fbced788"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "2c61da4e1fc8ab2a84d0fc3b6b4fd4ef"
server_key_iv: "96bfcf2693810f92322ea4403ec61a0b"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.970381 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd7533de6e3841e2933a3a72fbced788"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "2c61da4e1fc8ab2a84d0fc3b6b4fd4ef"
server_key_iv: "96bfcf2693810f92322ea4403ec61a0b"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.971698 4157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:19:19.972422 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:19.972612 4157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:19.972702 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "cd7533de6e3841e2933a3a72fbced788"
format_stamp: "Formatted at 2026-06-20 14:19:19 on dist-test-slave-h6n5"
server_key: "2c61da4e1fc8ab2a84d0fc3b6b4fd4ef"
server_key_iv: "96bfcf2693810f92322ea4403ec61a0b"
server_key_version: "encryptionkey@0"
I20260620 14:19:19.972776 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:20.016994 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:20.019634 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:20.019791 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:20.020049 4157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:20.020362 4157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:20.020433 4157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:20.020479 4157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:20.020502 4157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:20.026099 4157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46395
I20260620 14:19:20.026201 4285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46395 every 8 connection(s)
I20260620 14:19:20.026492 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:20.034683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4157
I20260620 14:19:20.034811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:20.035030 392 external_mini_cluster.cc:1428] Setting key 064bf06435e28100aefad6114165fec5
I20260620 14:19:20.036199 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:20.027208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37323 (local address 127.0.98.62:39837)
0620 14:19:20.027278 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:20.027281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:20.027512 (+ 231us) server_negotiation.cc:407] Connection header received
0620 14:19:20.028481 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:20.028482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:20.028492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:20.028545 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:20.029110 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.029607 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.030544 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.030720 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.030773 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:20.033044 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:20.033046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:20.033046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:20.033053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:20.034459 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.035026 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.035027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.035028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.035099 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.035385 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.035386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.035386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.035529 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:20.035628 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:20.035985 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:19:20.036081 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:20.036423 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:20.027200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39837 (local address 127.0.98.2:37323)
0620 14:19:20.027386 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:20.027421 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:20.028385 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:20.028596 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:20.028600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:20.028850 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:20.029039 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:20.029043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.029682 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.029682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:20.030471 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:20.030472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.030817 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.030818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:20.030938 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:20.031418 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:20.031426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:20.032954 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:20.034546 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:20.034553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:20.034555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:20.034945 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:20.035165 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:20.035166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:20.035166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:20.035285 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:20.035621 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:20.035627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:20.035811 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:19:20.035963 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:19:20.036151 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260620 14:19:20.036962 4286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39837
I20260620 14:19:20.037078 4286 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:20.037271 4286 heartbeater.cc:511] Master 127.0.98.62:39837 requested a full tablet report, sending...
I20260620 14:19:20.037689 3961 ts_manager.cc:194] Registered new tserver with Master: cd7533de6e3841e2933a3a72fbced788 (127.0.98.2:46395)
I20260620 14:19:20.038156 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37323
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:20.062594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39837
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:20.150126 4291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:20.150319 4291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:20.150341 4291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:20.151475 4291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:20.151542 4291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:20.151599 4291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:20.153075 4291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:20.153352 4291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:20.153579 4291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:20.158710 4296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:20.158758 4299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:20.158705 4297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:20.159083 4291 server_base.cc:1034] running on GCE node
I20260620 14:19:20.159368 4291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:20.159644 4291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:20.160785 4291 hybrid_clock.cc:648] HybridClock initialized: now 1781965160160759 us; error 47 us; skew 500 ppm
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:20.163349 4291 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:20.163895 4291 webserver.cc:460] Webserver started at http://127.0.98.3:45247/ using document root <none> and password file <none>
I20260620 14:19:20.164163 4291 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:20.164242 4291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:20.164381 4291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:20.165585 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "e34f21481b0a4ab8acfaddb588d6021e"
format_stamp: "Formatted at 2026-06-20 14:19:20 on dist-test-slave-h6n5"
server_key: "89523e45363ae08566a7e3a52c0c1b36"
server_key_iv: "148004e91497944db1f35c8d5734f307"
server_key_version: "encryptionkey@0"
I20260620 14:19:20.165895 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e34f21481b0a4ab8acfaddb588d6021e"
format_stamp: "Formatted at 2026-06-20 14:19:20 on dist-test-slave-h6n5"
server_key: "89523e45363ae08566a7e3a52c0c1b36"
server_key_iv: "148004e91497944db1f35c8d5734f307"
server_key_version: "encryptionkey@0"
I20260620 14:19:20.167178 4291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:20.167912 4306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:20.168131 4291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:19:20.168221 4291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "e34f21481b0a4ab8acfaddb588d6021e"
format_stamp: "Formatted at 2026-06-20 14:19:20 on dist-test-slave-h6n5"
server_key: "89523e45363ae08566a7e3a52c0c1b36"
server_key_iv: "148004e91497944db1f35c8d5734f307"
server_key_version: "encryptionkey@0"
I20260620 14:19:20.168274 4291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:20.181289 4291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:20.183122 4291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:20.183231 4291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:20.183391 4291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:20.183830 4291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:20.183889 4291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:20.183930 4291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:20.183959 4291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:20.189625 4291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38421
I20260620 14:19:20.189670 4419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38421 every 8 connection(s)
I20260620 14:19:20.189991 4291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:20.197664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4291
I20260620 14:19:20.197777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:20.197963 392 external_mini_cluster.cc:1428] Setting key a378146f1c10caaf4c8dc98f0626311c
I20260620 14:19:20.200039 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:20.190724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54361 (local address 127.0.98.62:39837)
0620 14:19:20.190795 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:20.190797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:20.191013 (+ 216us) server_negotiation.cc:407] Connection header received
0620 14:19:20.192205 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:20.192207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:20.192217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:20.192324 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:20.193196 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.193676 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.194726 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.194926 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.194984 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:20.197154 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:20.197155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:20.197155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:20.197162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:20.198904 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.199252 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.199253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.199253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.199306 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.199483 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.199483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.199483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.199648 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:20.199700 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:20.199918 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:19:20.199977 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:19:20.200209 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:20.190682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39837 (local address 127.0.98.3:54361)
0620 14:19:20.190878 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:20.190920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:20.191911 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:20.192477 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:20.192483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:20.192874 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:20.193111 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:20.193116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.193801 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.193802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:20.194652 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:20.194655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.195002 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.195003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:20.195101 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:20.195672 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:20.195680 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:20.197074 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:20.198963 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:20.198969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:20.198971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:20.199184 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:20.199351 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:20.199351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:20.199352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:20.199433 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:20.199691 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:20.199694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:20.199803 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:19:20.199890 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:19:20.200004 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260620 14:19:20.200667 4420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39837
I20260620 14:19:20.200767 4420 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:20.200944 4420 heartbeater.cc:511] Master 127.0.98.62:39837 requested a full tablet report, sending...
I20260620 14:19:20.201318 3961 ts_manager.cc:194] Registered new tserver with Master: e34f21481b0a4ab8acfaddb588d6021e (127.0.98.3:38421)
I20260620 14:19:20.201683 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54361
I20260620 14:19:20.209488 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:20.232501 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:20.224542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.0.98.1:33001)
0620 14:19:20.224715 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:20.224720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:20.224731 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:20.224757 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:20.224760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:20.224773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:20.224847 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:20.225430 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.226186 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.227008 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.227339 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.227402 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:20.229698 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:20.229705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:20.229705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:20.229712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:20.231401 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.231776 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.231779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.231779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.231848 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.232014 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.232014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.232014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.232132 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:20.232250 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:20.232315 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:19:20.232376 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:20.240190 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:20.233769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.0.98.62:39837)
0620 14:19:20.233833 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:20.233835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:20.233883 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:19:20.233908 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:20.233909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:20.233917 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:20.233965 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:20.234398 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.234858 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.235477 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:20.235736 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:20.235789 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:20.237799 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:20.237801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:20.237802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:20.237808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:20.239184 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.239466 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.239467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.239467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.239515 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:20.239735 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:20.239735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:20.239735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:20.239895 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:20.239990 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:20.240065 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:19:20.240124 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 14:19:20.240437 3961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46040
I20260620 14:19:20.240751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4023
I20260620 14:19:20.245543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4157
I20260620 14:19:20.250157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4291
I20260620 14:19:20.254274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 3929
2026-06-20T14:19:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1877 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[4432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[4432](info): set up 2 sockets
Jun 20 14:19:20 dist-test-slave-h6n5 krb5kdc[4432](info): commencing operation
krb5kdc: starting...
W20260620 14:19:21.287369 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:21 dist-test-slave-h6n5 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:21.407809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34481
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:21.488111 4445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:21.488337 4445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:21.488363 4445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:21.489627 4445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:21.489681 4445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:21.489696 4445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:21.489709 4445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:21.489722 4445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:21.491321 4445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:21.491629 4445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:21.491894 4445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:21.496749 4453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:21.496865 4445 server_base.cc:1034] running on GCE node
W20260620 14:19:21.496791 4450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:21.496722 4451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:21.497210 4445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:21.497509 4445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:21.498713 4445 hybrid_clock.cc:648] HybridClock initialized: now 1781965161498662 us; error 86 us; skew 500 ppm
Jun 20 14:19:21 dist-test-slave-h6n5 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:21.501593 4445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:21.502024 4445 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:21.502100 4445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:21.502256 4445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:21.503356 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "cdef413b8216cd99d0c1431437e5caa5"
server_key_iv: "50a027ab041b6e86bb9918b86454fb92"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.503749 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "cdef413b8216cd99d0c1431437e5caa5"
server_key_iv: "50a027ab041b6e86bb9918b86454fb92"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.505110 4445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:21.506124 4459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.506335 4445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:21.506446 4445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "cdef413b8216cd99d0c1431437e5caa5"
server_key_iv: "50a027ab041b6e86bb9918b86454fb92"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.506526 4445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:21.513929 4445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:21.516999 4445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:21.517160 4445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:21.521262 4445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34481
I20260620 14:19:21.521312 4511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34481 every 8 connection(s)
I20260620 14:19:21.521610 4445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:21.522351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4445
I20260620 14:19:21.522473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:21.522627 4512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:21.522653 392 external_mini_cluster.cc:1428] Setting key e7c56b11a83ce7b3faeb693e1dcfe08f
I20260620 14:19:21.526078 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb: Bootstrap starting.
I20260620 14:19:21.526801 4512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:21.527103 4512 log.cc:824] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:21.527907 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb: No bootstrap required, opened a new log
I20260620 14:19:21.529716 4512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34481 } }
I20260620 14:19:21.529860 4512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:21.529883 4512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff8dc2dfd4a454dad56561ac39bb4bb, State: Initialized, Role: FOLLOWER
I20260620 14:19:21.530026 4512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34481 } }
I20260620 14:19:21.530102 4512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Jun 20 14:19:21 dist-test-slave-h6n5 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:21.530159 4512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:21.530210 4512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:21.530782 4512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34481 } }
I20260620 14:19:21.530905 4512 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff8dc2dfd4a454dad56561ac39bb4bb; no voters:
I20260620 14:19:21.531101 4512 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:21.531131 4517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:21.531250 4517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [term 1 LEADER]: Becoming Leader. State: Replica: 7ff8dc2dfd4a454dad56561ac39bb4bb, State: Running, Role: LEADER
I20260620 14:19:21.531388 4517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34481 } }
I20260620 14:19:21.531390 4512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:21.531925 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34481 } } }
I20260620 14:19:21.532009 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:21.532266 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff8dc2dfd4a454dad56561ac39bb4bb. Latest consensus state: current_term: 1 leader_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff8dc2dfd4a454dad56561ac39bb4bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34481 } } }
I20260620 14:19:21.533149 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb [sys.catalog]: This master's current role is: LEADER
W20260620 14:19:21.533381 4533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:19:21.533449 4533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:19:21.533538 4523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:21.533723 4523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:21.534559 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:21.523320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.0.98.62:34481)
0620 14:19:21.523492 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:21.523502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:21.523538 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:19:21.524234 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:21.524244 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:21.524313 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:21.524487 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:21.525111 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.526518 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.527481 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.528009 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.528141 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:21.530650 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:21.530655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:21.530655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:21.530667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:21.532864 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:21.533309 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:21.533312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:21.533313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:21.533426 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:21.533631 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:21.533631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:21.533632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:21.533922 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:21.534063 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:21.534218 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:19:21.534367 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260620 14:19:21.535367 4523 catalog_manager.cc:1347] Generated new cluster ID: 67567c22670e4b5a9ec447064fffd35a
I20260620 14:19:21.535418 4523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:21.551241 4523 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:21.552081 4523 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:21.557276 4523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb: Generated new TSK 0
I20260620 14:19:21.557468 4523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:21.586788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34481
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:21.668679 4538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:21.668900 4538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:21.668924 4538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:21.670125 4538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:21.670168 4538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:21.670195 4538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:21.671787 4538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:21.672034 4538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:21.672284 4538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:21.677541 4546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:21.677541 4543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:21.677559 4544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:21.677832 4538 server_base.cc:1034] running on GCE node
I20260620 14:19:21.678011 4538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:21.678231 4538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:21.679391 4538 hybrid_clock.cc:648] HybridClock initialized: now 1781965161679375 us; error 28 us; skew 500 ppm
Jun 20 14:19:21 dist-test-slave-h6n5 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:21.681907 4538 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:21.682279 4538 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:21.682363 4538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:21.682516 4538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:21.683656 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "ffbf8769aee44a1cac4d21bba5916fa3"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "251fab9de9948646d80826783ed13501"
server_key_iv: "c9e6d405d6bed7b41acde3879d357fbc"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.684023 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ffbf8769aee44a1cac4d21bba5916fa3"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "251fab9de9948646d80826783ed13501"
server_key_iv: "c9e6d405d6bed7b41acde3879d357fbc"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.685359 4538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:21.686156 4552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.686372 4538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:21.686479 4538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "ffbf8769aee44a1cac4d21bba5916fa3"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "251fab9de9948646d80826783ed13501"
server_key_iv: "c9e6d405d6bed7b41acde3879d357fbc"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.686560 4538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:21.734155 4538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:21.736485 4538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:21.736668 4538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:21.736940 4538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:21.737287 4538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:21.737349 4538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.737392 4538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:21.737442 4538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.743855 4538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34207
I20260620 14:19:21.743940 4665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34207 every 8 connection(s)
I20260620 14:19:21.744230 4538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:19:21 dist-test-slave-h6n5 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:21.752290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4538
I20260620 14:19:21.752414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:21.752631 392 external_mini_cluster.cc:1428] Setting key 0f3581b7c3beac6cf2220c5214fb1f2b
I20260620 14:19:21.753672 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:21.744958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55315 (local address 127.0.98.62:34481)
0620 14:19:21.745005 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:21.745007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:21.745248 (+ 241us) server_negotiation.cc:407] Connection header received
0620 14:19:21.745768 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:21.745769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:21.745781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:21.745840 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:21.746751 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.747264 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.748308 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.748527 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.748615 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:21.750720 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:21.750722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:21.750722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:21.750729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:21.752195 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:21.752672 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:21.752674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:21.752674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:21.752760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:21.753025 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:21.753026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:21.753027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:21.753144 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:21.753215 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:21.753493 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:19:21.753583 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:21.753888 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:21.744932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34481 (local address 127.0.98.1:55315)
0620 14:19:21.745119 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:21.745156 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:21.745658 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:21.745915 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:21.745919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:21.746178 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:21.746646 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:21.746650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.747374 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.747375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:21.748203 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:21.748205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.748624 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.748625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:21.748747 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:21.749257 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:21.749266 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:21.750606 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:21.752280 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:21.752287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:21.752289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:21.752583 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:21.752836 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:21.752837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:21.752838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:21.752955 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:21.753216 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:21.753223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:21.753364 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:19:21.753471 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:19:21.753630 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260620 14:19:21.754946 4666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34481
I20260620 14:19:21.755061 4666 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:21.755215 4666 heartbeater.cc:511] Master 127.0.98.62:34481 requested a full tablet report, sending...
I20260620 14:19:21.755893 4476 ts_manager.cc:194] Registered new tserver with Master: ffbf8769aee44a1cac4d21bba5916fa3 (127.0.98.1:34207)
I20260620 14:19:21.756850 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55315
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:21.779071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34481
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:21.862658 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:21.862869 4671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:21.862892 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:21.864078 4671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:21.864120 4671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:21.864146 4671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:21.865657 4671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:21.865909 4671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:21.866253 4671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:21.871417 4677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:21.871438 4676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:21.871721 4679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:21.871696 4671 server_base.cc:1034] running on GCE node
I20260620 14:19:21.871976 4671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:21.872225 4671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:21.873392 4671 hybrid_clock.cc:648] HybridClock initialized: now 1781965161873391 us; error 47 us; skew 500 ppm
Jun 20 14:19:21 dist-test-slave-h6n5 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:21.876077 4671 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:21.876427 4671 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:21.876477 4671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:21.876610 4671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:21.877672 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8b695af87694d198e609f8124dd79c5"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "bfa0560d7bd9df35e048ab392e17bf64"
server_key_iv: "a89b129f28fcfb6c21e6731cf532ee37"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.878019 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8b695af87694d198e609f8124dd79c5"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "bfa0560d7bd9df35e048ab392e17bf64"
server_key_iv: "a89b129f28fcfb6c21e6731cf532ee37"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.879331 4671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:21.880137 4685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.880321 4671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.880409 4671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "c8b695af87694d198e609f8124dd79c5"
format_stamp: "Formatted at 2026-06-20 14:19:21 on dist-test-slave-h6n5"
server_key: "bfa0560d7bd9df35e048ab392e17bf64"
server_key_iv: "a89b129f28fcfb6c21e6731cf532ee37"
server_key_version: "encryptionkey@0"
I20260620 14:19:21.880484 4671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:21.893520 4671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:21.895678 4671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:21.895798 4671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:21.896077 4671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:21.896399 4671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:21.896457 4671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.896499 4671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:21.896528 4671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:21.902524 4671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46603
I20260620 14:19:21.902612 4798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46603 every 8 connection(s)
I20260620 14:19:21.902884 4671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:21.904132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4671
I20260620 14:19:21.904220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:21.904426 392 external_mini_cluster.cc:1428] Setting key 958a7c2751f3f51fca628113043d954e
Jun 20 14:19:21 dist-test-slave-h6n5 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:19:21.914064 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:21.903678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50809 (local address 127.0.98.62:34481)
0620 14:19:21.903745 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:21.903747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:21.903996 (+ 249us) server_negotiation.cc:407] Connection header received
0620 14:19:21.904644 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:21.904646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:21.904656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:21.904710 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:21.905758 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.906543 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.907601 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.907929 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.907997 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:21.910378 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:21.910379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:21.910380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:21.910388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:21.912115 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:21.912649 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:21.912650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:21.912650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:21.912740 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:21.913105 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:21.913107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:21.913107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:21.913280 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:21.913357 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:21.913803 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 14:19:21.913932 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:19:21.914250 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:21.903664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34481 (local address 127.0.98.2:50809)
0620 14:19:21.903862 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:21.903900 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:21.904511 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:21.904791 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:21.904795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:21.905162 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:21.905661 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:21.905666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.906647 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.906648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:21.907481 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:21.907483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:21.908029 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:21.908031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:21.908145 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:21.908720 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:21.908770 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:21.910287 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:21.912213 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:21.912219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:21.912222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:21.912527 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:21.912814 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:21.912815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:21.912834 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:21.913010 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:21.913346 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:21.913353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:21.913624 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 14:19:21.913749 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:19:21.913957 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260620 14:19:21.915958 4799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:19:21.916081 4799 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:21.916258 4799 heartbeater.cc:511] Master 127.0.98.62:34481 requested a full tablet report, sending...
I20260620 14:19:21.916756 4476 ts_manager.cc:194] Registered new tserver with Master: c8b695af87694d198e609f8124dd79c5 (127.0.98.2:46603)
I20260620 14:19:21.917387 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:21.932662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34481
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:22.013991 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:22.014191 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:22.014213 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:22.015293 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:22.015324 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:22.015350 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:22.016826 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:22.017053 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:22.017236 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:22.022239 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.022239 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.022258 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:22.022310 4804 server_base.cc:1034] running on GCE node
I20260620 14:19:22.022684 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:22.022943 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:22.024106 4804 hybrid_clock.cc:648] HybridClock initialized: now 1781965162024093 us; error 48 us; skew 500 ppm
Jun 20 14:19:22 dist-test-slave-h6n5 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:22.026666 4804 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:22.027190 4804 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:22.027272 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:22.027437 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:22.028609 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "645efb3ee42c4f21b2e96e3a7988e1c8"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "7a4a3bed383f9139c8c16350fa62681c"
server_key_iv: "673f9e3606b7bcbf9a072ec0ec06b3d8"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.028978 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "645efb3ee42c4f21b2e96e3a7988e1c8"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "7a4a3bed383f9139c8c16350fa62681c"
server_key_iv: "673f9e3606b7bcbf9a072ec0ec06b3d8"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.030253 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:19:22.030997 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.031198 4804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:22.031293 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "645efb3ee42c4f21b2e96e3a7988e1c8"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "7a4a3bed383f9139c8c16350fa62681c"
server_key_iv: "673f9e3606b7bcbf9a072ec0ec06b3d8"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.031380 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:22.050788 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:22.053004 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:22.053181 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:22.053447 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:22.053771 4804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:22.053825 4804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.053865 4804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:22.053895 4804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.060065 4804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42509
I20260620 14:19:22.060101 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42509 every 8 connection(s)
I20260620 14:19:22.060365 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:19:22 dist-test-slave-h6n5 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:22.067871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4804
I20260620 14:19:22.067965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:22.068164 392 external_mini_cluster.cc:1428] Setting key 506011c71215bb13e2eb497ad0484236
I20260620 14:19:22.069607 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.061094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59685 (local address 127.0.98.62:34481)
0620 14:19:22.061160 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.061163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.061379 (+ 216us) server_negotiation.cc:407] Connection header received
0620 14:19:22.061833 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.061834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.061844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.061897 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.062769 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.063255 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.064297 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.064482 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.064539 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.066547 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.066548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.066548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:22.066555 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.068147 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:22.068543 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:22.068544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:22.068545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:22.068612 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:22.068939 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:22.068939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:22.068940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:22.069088 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.069155 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.069470 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.069544 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:19:22.069748 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.061074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34481 (local address 127.0.98.3:59685)
0620 14:19:22.061264 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.061300 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.061755 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.061962 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.061965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.062217 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.062693 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.062698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.063335 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.063336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.064223 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.064225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.064563 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.064564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.064664 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.065118 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:22.065125 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.066443 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.068232 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:22.068240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:22.068242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:22.068486 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:22.068725 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:22.068726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:22.068726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:22.068865 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:22.069195 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.069210 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.069342 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.069440 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.069559 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260620 14:19:22.070215 4932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34481
I20260620 14:19:22.070291 4932 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:22.070456 4932 heartbeater.cc:511] Master 127.0.98.62:34481 requested a full tablet report, sending...
I20260620 14:19:22.070828 4476 ts_manager.cc:194] Registered new tserver with Master: 645efb3ee42c4f21b2e96e3a7988e1c8 (127.0.98.3:42509)
I20260620 14:19:22.071153 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59685
I20260620 14:19:22.079965 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:19:22.087158 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.081199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.0.98.62:34481)
0620 14:19:22.081257 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.081259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.081374 (+ 115us) server_negotiation.cc:407] Connection header received
0620 14:19:22.081414 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.081415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.081423 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.081473 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.082008 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.082483 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.083277 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.083461 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.083516 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.084776 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.084778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.084778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:22.084784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.086229 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:22.086513 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:22.086513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:22.086514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:22.086560 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:22.086700 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:22.086701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:22.086701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:22.086855 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.086927 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.087014 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.087069 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:19:22.088732 4476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:22.090409 4476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:22.097096 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.092897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.0.98.2:46603)
0620 14:19:22.093162 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.093167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.093177 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:22.093219 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.093221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.093238 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.093332 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:22.093516 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.094677 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.095988 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.096723 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.096805 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.096864 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.096965 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.097008 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":76,"threads_started":1}
I20260620 14:19:22.097106 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.092709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46603 (local address 127.0.0.1:54728)
0620 14:19:22.092888 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.092922 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.092988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.093346 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.093349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.093352 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:22.093426 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.093428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.094807 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.094808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.095892 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.095899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.096808 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.096808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.096862 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.096873 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.096932 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.097005 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":48,"thread_start_us":80,"threads_started":1}
I20260620 14:19:22.097291 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.092985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.0.98.3:42509)
0620 14:19:22.093476 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.093480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.093489 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:22.093685 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.093687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.093702 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.093804 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:22.093915 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.094591 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.096187 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.096877 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.096972 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.097027 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.097166 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.097214 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":68,"threads_started":1}
I20260620 14:19:22.097487 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.092725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42509 (local address 127.0.0.1:35184)
0620 14:19:22.093247 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.093257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.093304 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.093801 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.093802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.093804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:22.093848 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.093850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.094692 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.094693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.096102 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.096104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.096981 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.096982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.097031 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.097034 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.097084 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.097120 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":55,"thread_start_us":130,"threads_started":1}
I20260620 14:19:22.098033 4733 tablet_service.cc:1467] Processing CreateTablet for tablet b69da16eead04370aeb17827a5074700 (DEFAULT_TABLE table=test-table [id=309e2eae82f7408cb862fb94cce7a16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:22.098381 4733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69da16eead04370aeb17827a5074700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:22.098518 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.092709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34207 (local address 127.0.0.1:41998)
0620 14:19:22.092865 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.092922 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.092988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.093482 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.093483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.093485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:22.093525 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.093527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.094766 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.094767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.096092 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.096094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.098277 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.098278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.098349 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.098354 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.098406 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.098449 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":86,"thread_start_us":52,"threads_started":1}
I20260620 14:19:22.098531 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.093076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.0.98.1:34207)
0620 14:19:22.093234 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.093238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.093246 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:22.093276 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.093278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.093290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.093395 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:22.093603 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.094677 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.097062 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.098172 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.098272 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.098327 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.098419 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.098454 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260620 14:19:22.101025 4949 tablet_bootstrap.cc:492] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Bootstrap starting.
I20260620 14:19:22.101343 4866 tablet_service.cc:1467] Processing CreateTablet for tablet b69da16eead04370aeb17827a5074700 (DEFAULT_TABLE table=test-table [id=309e2eae82f7408cb862fb94cce7a16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:22.101631 4949 tablet_bootstrap.cc:654] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:22.101634 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69da16eead04370aeb17827a5074700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:22.102162 4949 log.cc:824] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:22.102896 4949 tablet_bootstrap.cc:492] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: No bootstrap required, opened a new log
I20260620 14:19:22.102981 4949 ts_tablet_manager.cc:1397] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:22.104537 4951 tablet_bootstrap.cc:492] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: Bootstrap starting.
I20260620 14:19:22.104966 4949 raft_consensus.cc:348] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.105015 4600 tablet_service.cc:1467] Processing CreateTablet for tablet b69da16eead04370aeb17827a5074700 (DEFAULT_TABLE table=test-table [id=309e2eae82f7408cb862fb94cce7a16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:22.105142 4949 raft_consensus.cc:374] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:22.105185 4949 raft_consensus.cc:729] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b695af87694d198e609f8124dd79c5, State: Initialized, Role: FOLLOWER
I20260620 14:19:22.105279 4951 tablet_bootstrap.cc:654] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:22.105338 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69da16eead04370aeb17827a5074700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:22.105350 4949 consensus_queue.cc:260] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.105686 4799 heartbeater.cc:503] Master 127.0.98.62:34481 was elected leader, sending a full tablet report...
I20260620 14:19:22.105705 4949 ts_tablet_manager.cc:1428] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:19:22.105871 4951 log.cc:824] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:22.106992 4951 tablet_bootstrap.cc:492] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: No bootstrap required, opened a new log
I20260620 14:19:22.107064 4951 ts_tablet_manager.cc:1397] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:19:22.108404 4954 tablet_bootstrap.cc:492] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3: Bootstrap starting.
I20260620 14:19:22.108783 4951 raft_consensus.cc:348] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.108952 4951 raft_consensus.cc:374] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:22.108989 4951 raft_consensus.cc:729] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645efb3ee42c4f21b2e96e3a7988e1c8, State: Initialized, Role: FOLLOWER
I20260620 14:19:22.109144 4954 tablet_bootstrap.cc:654] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:22.109114 4951 consensus_queue.cc:260] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.109416 4951 ts_tablet_manager.cc:1428] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:19:22.109431 4932 heartbeater.cc:503] Master 127.0.98.62:34481 was elected leader, sending a full tablet report...
I20260620 14:19:22.109732 4954 log.cc:824] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:22.110437 4954 tablet_bootstrap.cc:492] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3: No bootstrap required, opened a new log
I20260620 14:19:22.110489 4954 ts_tablet_manager.cc:1397] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:22.111948 4954 raft_consensus.cc:348] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.112079 4954 raft_consensus.cc:374] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:22.112100 4954 raft_consensus.cc:729] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbf8769aee44a1cac4d21bba5916fa3, State: Initialized, Role: FOLLOWER
I20260620 14:19:22.112221 4954 consensus_queue.cc:260] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.112509 4666 heartbeater.cc:503] Master 127.0.98.62:34481 was elected leader, sending a full tablet report...
I20260620 14:19:22.112519 4954 ts_tablet_manager.cc:1428] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:22.141685 4955 raft_consensus.cc:482] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:22.141808 4955 raft_consensus.cc:504] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.142148 4955 leader_election.cc:288] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8b695af87694d198e609f8124dd79c5 (127.0.98.2:46603), ffbf8769aee44a1cac4d21bba5916fa3 (127.0.98.1:34207)
I20260620 14:19:22.146157 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.142439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37307 (local address 127.0.98.1:34207)
0620 14:19:22.142492 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.142494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.142603 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:19:22.142631 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.142631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.142643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.142705 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:22.142882 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.143667 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.145216 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.145769 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.145854 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.145866 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.146047 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.146088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:19:22.146157 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.142361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33329 (local address 127.0.98.2:46603)
0620 14:19:22.142423 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.142426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.142437 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:22.142542 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.142543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.142554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.142636 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:22.142883 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.143658 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.145216 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.145792 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.145862 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.145871 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.146047 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.146089 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:19:22.146145 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.142306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46603 (local address 127.0.98.3:33329)
0620 14:19:22.142368 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.142383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.142430 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.142660 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.142660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.142662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:22.142749 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.142751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.143770 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.143771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.145110 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.145112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.145874 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.145875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.145971 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.145973 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.146025 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.146056 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":76}
I20260620 14:19:22.146159 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.142343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34207 (local address 127.0.98.3:37307)
0620 14:19:22.142522 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.142531 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.142582 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.142717 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.142718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.142719 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:22.142779 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.142780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.143756 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.143758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.145110 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:22.145112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.145874 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.145875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:22.145945 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.145952 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.146014 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.146054 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":54,"thread_start_us":114,"threads_started":1}
I20260620 14:19:22.146524 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69da16eead04370aeb17827a5074700" candidate_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8b695af87694d198e609f8124dd79c5" is_pre_election: true
I20260620 14:19:22.146554 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69da16eead04370aeb17827a5074700" candidate_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" is_pre_election: true
I20260620 14:19:22.146709 4620 raft_consensus.cc:2454] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645efb3ee42c4f21b2e96e3a7988e1c8 in term 0.
I20260620 14:19:22.146693 4753 raft_consensus.cc:2454] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645efb3ee42c4f21b2e96e3a7988e1c8 in term 0.
I20260620 14:19:22.146962 4821 leader_election.cc:302] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645efb3ee42c4f21b2e96e3a7988e1c8, c8b695af87694d198e609f8124dd79c5; no voters:
I20260620 14:19:22.147114 4955 raft_consensus.cc:2781] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:22.147169 4955 raft_consensus.cc:482] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:22.147197 4955 raft_consensus.cc:3037] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:22.147938 4955 raft_consensus.cc:504] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.148089 4955 leader_election.cc:288] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [CANDIDATE]: Term 1 election: Requested vote from peers c8b695af87694d198e609f8124dd79c5 (127.0.98.2:46603), ffbf8769aee44a1cac4d21bba5916fa3 (127.0.98.1:34207)
I20260620 14:19:22.148224 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69da16eead04370aeb17827a5074700" candidate_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8b695af87694d198e609f8124dd79c5"
I20260620 14:19:22.148234 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69da16eead04370aeb17827a5074700" candidate_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbf8769aee44a1cac4d21bba5916fa3"
I20260620 14:19:22.148304 4620 raft_consensus.cc:3037] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:22.148304 4753 raft_consensus.cc:3037] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:22.149170 4620 raft_consensus.cc:2454] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645efb3ee42c4f21b2e96e3a7988e1c8 in term 1.
I20260620 14:19:22.149170 4753 raft_consensus.cc:2454] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645efb3ee42c4f21b2e96e3a7988e1c8 in term 1.
I20260620 14:19:22.149379 4820 leader_election.cc:302] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645efb3ee42c4f21b2e96e3a7988e1c8, ffbf8769aee44a1cac4d21bba5916fa3; no voters:
I20260620 14:19:22.149497 4955 raft_consensus.cc:2781] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:22.149652 4955 raft_consensus.cc:686] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 1 LEADER]: Becoming Leader. State: Replica: 645efb3ee42c4f21b2e96e3a7988e1c8, State: Running, Role: LEADER
I20260620 14:19:22.149785 4955 consensus_queue.cc:237] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } }
I20260620 14:19:22.150656 4476 catalog_manager.cc:5495] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 645efb3ee42c4f21b2e96e3a7988e1c8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "645efb3ee42c4f21b2e96e3a7988e1c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42509 } health_report { overall_health: HEALTHY } } }
W20260620 14:19:22.153410 4800 tablet.cc:2369] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:22.169550 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.167002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.98.3:42509)
0620 14:19:22.167046 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.167048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.167058 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:22.167133 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.167134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.167155 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.167258 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:22.167417 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.167957 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.168738 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:22.168946 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:22.169031 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:22.169214 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:22.169316 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:22.169410 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.169429 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.169448 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:19:22.171227 4620 raft_consensus.cc:1261] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 1 FOLLOWER]: Refusing update from remote peer 645efb3ee42c4f21b2e96e3a7988e1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:22.171227 4753 raft_consensus.cc:1261] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 1 FOLLOWER]: Refusing update from remote peer 645efb3ee42c4f21b2e96e3a7988e1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:22.171581 4955 consensus_queue.cc:1035] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffbf8769aee44a1cac4d21bba5916fa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:22.171808 4959 consensus_queue.cc:1035] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b695af87694d198e609f8124dd79c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:22.176388 4475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:22.176465 4475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:22.177184 4475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ff8dc2dfd4a454dad56561ac39bb4bb: Sending DeleteTablet for 3 replicas of tablet b69da16eead04370aeb17827a5074700
I20260620 14:19:22.177544 4733 tablet_service.cc:1514] Processing DeleteTablet for tablet b69da16eead04370aeb17827a5074700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54728
I20260620 14:19:22.177769 4972 tablet_replica.cc:330] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: stopping tablet replica
I20260620 14:19:22.177762 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet b69da16eead04370aeb17827a5074700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35184
I20260620 14:19:22.177912 4972 raft_consensus.cc:2229] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:22.177974 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet b69da16eead04370aeb17827a5074700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41998
I20260620 14:19:22.178068 4972 raft_consensus.cc:2258] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:22.178268 4974 tablet_replica.cc:330] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3: stopping tablet replica
I20260620 14:19:22.178397 4974 raft_consensus.cc:2229] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:22.178429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4538
I20260620 14:19:22.178548 4974 raft_consensus.cc:2258] T b69da16eead04370aeb17827a5074700 P ffbf8769aee44a1cac4d21bba5916fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:22.178646 4972 ts_tablet_manager.cc:1905] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:22.180335 4972 ts_tablet_manager.cc:1918] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:22.180438 4972 log.cc:1196] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/b69da16eead04370aeb17827a5074700
I20260620 14:19:22.180732 4972 ts_tablet_manager.cc:1939] T b69da16eead04370aeb17827a5074700 P c8b695af87694d198e609f8124dd79c5: Deleting consensus metadata
I20260620 14:19:22.181744 4462 catalog_manager.cc:4841] TS c8b695af87694d198e609f8124dd79c5 (127.0.98.2:46603): tablet b69da16eead04370aeb17827a5074700 (table test-table [id=309e2eae82f7408cb862fb94cce7a16a]) successfully deleted
W20260620 14:19:22.184609 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34207 (error 108)
I20260620 14:19:22.184871 4973 tablet_replica.cc:330] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: stopping tablet replica
I20260620 14:19:22.184990 4973 raft_consensus.cc:2229] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:22.185111 4973 raft_consensus.cc:2258] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:22.185108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4671
I20260620 14:19:22.185125 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.185002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34207 (local address 127.0.0.1:42000)
0620 14:19:22.185047 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.185087 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:19:22.186132 4973 ts_tablet_manager.cc:1905] T b69da16eead04370aeb17827a5074700 P 645efb3ee42c4f21b2e96e3a7988e1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:19:22.187655 4461 catalog_manager.cc:4584] TS ffbf8769aee44a1cac4d21bba5916fa3 (127.0.98.1:34207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b69da16eead04370aeb17827a5074700: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34207: connect: Connection refused (error 111)
I20260620 14:19:22.193032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4804
I20260620 14:19:22.213496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4445
2026-06-20T14:19:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1960 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:22Z Disabled control of system clock
I20260620 14:19:22.242252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39401
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:19:22.323453 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:22.323763 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:22.323809 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:22.325184 4979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:22.325227 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:22.325241 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:22.325254 4979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:22.325302 4979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:22.326682 4979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:22.326944 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:22.327241 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:22.332053 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.332113 4985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.332116 4984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:22.332362 4979 server_base.cc:1034] running on GCE node
I20260620 14:19:22.332615 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:22.332906 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:22.334050 4979 hybrid_clock.cc:648] HybridClock initialized: now 1781965162334030 us; error 37 us; skew 500 ppm
I20260620 14:19:22.335314 4979 webserver.cc:460] Webserver started at http://127.0.98.62:43175/ using document root <none> and password file <none>
I20260620 14:19:22.335618 4979 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:22.335671 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:22.335814 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:22.336895 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "46940e70a48e499e8873dc04265c4117"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "a4830bad0b5e06f015669a2bc3251b99"
server_key_iv: "105a9684a53289c8aa0020e37479cde5"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.337265 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "46940e70a48e499e8873dc04265c4117"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "a4830bad0b5e06f015669a2bc3251b99"
server_key_iv: "105a9684a53289c8aa0020e37479cde5"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.338572 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:22.339298 4993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.339505 4979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:22.339650 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "46940e70a48e499e8873dc04265c4117"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "a4830bad0b5e06f015669a2bc3251b99"
server_key_iv: "105a9684a53289c8aa0020e37479cde5"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.339727 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:22.342985 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:22.343282 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:22.343431 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:22.347539 4979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39401
I20260620 14:19:22.347618 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39401 every 8 connection(s)
I20260620 14:19:22.347957 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:22.348660 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:22.351347 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117: Bootstrap starting.
I20260620 14:19:22.351946 5046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:22.352188 5046 log.cc:824] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:22.352814 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117: No bootstrap required, opened a new log
I20260620 14:19:22.354337 5046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46940e70a48e499e8873dc04265c4117" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260620 14:19:22.354473 5046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:22.354496 5046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46940e70a48e499e8873dc04265c4117, State: Initialized, Role: FOLLOWER
I20260620 14:19:22.354633 5046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46940e70a48e499e8873dc04265c4117" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260620 14:19:22.354708 5046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:22.354732 5046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:22.354765 5046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:22.356570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 4979
I20260620 14:19:22.356647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:22.356765 392 external_mini_cluster.cc:1428] Setting key 8ea9218721742cda3f4cb001e90f31b3
I20260620 14:19:22.358829 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.357233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.0.98.62:39401)
0620 14:19:22.357462 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.357471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.357503 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:19:22.357596 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.357602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.357637 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.357773 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.358382 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.358385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.358386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.358390 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.358435 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.358445 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.358516 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.358595 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.358654 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260620 14:19:22.370154 5046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46940e70a48e499e8873dc04265c4117" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260620 14:19:22.370285 5046 leader_election.cc:302] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46940e70a48e499e8873dc04265c4117; no voters:
I20260620 14:19:22.370518 5046 leader_election.cc:288] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:22.370641 5051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:22.370832 5046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:22.370832 5051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [term 1 LEADER]: Becoming Leader. State: Replica: 46940e70a48e499e8873dc04265c4117, State: Running, Role: LEADER
I20260620 14:19:22.371212 5051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46940e70a48e499e8873dc04265c4117" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260620 14:19:22.371644 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46940e70a48e499e8873dc04265c4117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46940e70a48e499e8873dc04265c4117" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } } }
I20260620 14:19:22.371740 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:22.371657 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46940e70a48e499e8873dc04265c4117. Latest consensus state: current_term: 1 leader_uuid: "46940e70a48e499e8873dc04265c4117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46940e70a48e499e8873dc04265c4117" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } } }
I20260620 14:19:22.371791 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:22.371974 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:22.372493 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:22.375823 5060 catalog_manager.cc:1347] Generated new cluster ID: 7927283944374edb82f3389119468771
I20260620 14:19:22.375888 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:22.392441 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:22.392642 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:22.403666 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117: Generated new TSK 0
I20260620 14:19:22.403918 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:22.409183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:19:22.501734 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:22.502034 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:22.502067 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:22.503582 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:22.503644 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:22.503688 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:22.505105 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:22.505383 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:22.505692 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:22.511746 5070 server_base.cc:1034] running on GCE node
W20260620 14:19:22.511720 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.511837 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.511720 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:22.512162 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:22.512416 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:22.513577 5070 hybrid_clock.cc:648] HybridClock initialized: now 1781965162513563 us; error 30 us; skew 500 ppm
I20260620 14:19:22.514786 5070 webserver.cc:460] Webserver started at http://127.0.98.1:43761/ using document root <none> and password file <none>
I20260620 14:19:22.515007 5070 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:22.515048 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:22.515133 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:22.516203 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "c26045cb5754407080414b7a0a145bf2"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "5765f602a03cd459d72179e8af6cb7f7"
server_key_iv: "fd6705b3f796c7cb1c52f246f89db441"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.516505 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c26045cb5754407080414b7a0a145bf2"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "5765f602a03cd459d72179e8af6cb7f7"
server_key_iv: "fd6705b3f796c7cb1c52f246f89db441"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.517750 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:22.518417 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.518589 5070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:22.518653 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "c26045cb5754407080414b7a0a145bf2"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "5765f602a03cd459d72179e8af6cb7f7"
server_key_iv: "fd6705b3f796c7cb1c52f246f89db441"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.518712 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:22.521898 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:22.522104 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:22.522194 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:22.522336 5070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:22.522569 5070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:22.522598 5070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.522617 5070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:22.522629 5070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.528788 5070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37141
I20260620 14:19:22.528816 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37141 every 8 connection(s)
I20260620 14:19:22.529181 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:22.532246 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.529895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59597 (local address 127.0.98.62:39401)
0620 14:19:22.529958 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.529960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.530210 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:19:22.530733 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.530735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.530742 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.530842 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.531986 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.531988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.531988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.531993 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.532009 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.532013 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.532073 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.532161 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.532178 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:22.532466 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.529915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39401 (local address 127.0.98.1:59597)
0620 14:19:22.530096 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.530129 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.530618 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.530918 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.530921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.530928 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.531841 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.531842 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.531869 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.531894 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.532071 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.532075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.532078 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.532172 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.532230 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260620 14:19:22.532964 5198 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39401
I20260620 14:19:22.533061 5198 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:22.533203 5198 heartbeater.cc:511] Master 127.0.98.62:39401 requested a full tablet report, sending...
I20260620 14:19:22.533713 5010 ts_manager.cc:194] Registered new tserver with Master: c26045cb5754407080414b7a0a145bf2 (127.0.98.1:37141)
I20260620 14:19:22.534837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5070
I20260620 14:19:22.534958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:22.535120 392 external_mini_cluster.cc:1428] Setting key 7d4fdc288a16fe73fd0b53c285469ddd
I20260620 14:19:22.536206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:19:22.623157 5201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:22.623400 5201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:22.623425 5201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:22.624637 5201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:22.624676 5201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:22.624702 5201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:22.626063 5201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:22.626288 5201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:22.626575 5201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:22.631594 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.631606 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:22.631646 5201 server_base.cc:1034] running on GCE node
W20260620 14:19:22.631605 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:22.631965 5201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:22.632159 5201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:22.633323 5201 hybrid_clock.cc:648] HybridClock initialized: now 1781965162633305 us; error 31 us; skew 500 ppm
I20260620 14:19:22.634804 5201 webserver.cc:460] Webserver started at http://127.0.98.2:41677/ using document root <none> and password file <none>
I20260620 14:19:22.635051 5201 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:22.635121 5201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:22.635270 5201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:22.636368 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "3667c79e1ab542548327bc722bd4ab41"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "4a31e57bcf1bf271cfbfe2b61517ce73"
server_key_iv: "52060f01344e8fa1ce125973f8ccfe4e"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.636724 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3667c79e1ab542548327bc722bd4ab41"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "4a31e57bcf1bf271cfbfe2b61517ce73"
server_key_iv: "52060f01344e8fa1ce125973f8ccfe4e"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.638006 5201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:19:22.638687 5215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.638854 5201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:22.638978 5201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "3667c79e1ab542548327bc722bd4ab41"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "4a31e57bcf1bf271cfbfe2b61517ce73"
server_key_iv: "52060f01344e8fa1ce125973f8ccfe4e"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.639063 5201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:22.642684 5201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:22.642949 5201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:22.643090 5201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:22.643286 5201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:22.643652 5201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:22.643709 5201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.643752 5201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:22.643778 5201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.649883 5201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40451
I20260620 14:19:22.649924 5328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40451 every 8 connection(s)
I20260620 14:19:22.650238 5201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:22.650651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5201
I20260620 14:19:22.650751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:22.650946 392 external_mini_cluster.cc:1428] Setting key 601bcf51e531d85be595c89c3f3de459
I20260620 14:19:22.651916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:19:22.654088 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.651011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34915 (local address 127.0.98.62:39401)
0620 14:19:22.651076 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.651078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.651314 (+ 236us) server_negotiation.cc:407] Connection header received
0620 14:19:22.651932 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.651933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.651940 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.652006 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.653820 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.653821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.653821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.653825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.653841 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.653846 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.653914 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.654021 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.654035 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:22.654328 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.650998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39401 (local address 127.0.98.2:34915)
0620 14:19:22.651194 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.651239 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.651826 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.652091 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.652095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.652106 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.653639 (+ 1533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.653640 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.653681 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.653720 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.653935 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.653940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.653942 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.654032 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.654097 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260620 14:19:22.654760 5329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39401
I20260620 14:19:22.654871 5329 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:22.655023 5329 heartbeater.cc:511] Master 127.0.98.62:39401 requested a full tablet report, sending...
I20260620 14:19:22.655421 5010 ts_manager.cc:194] Registered new tserver with Master: 3667c79e1ab542548327bc722bd4ab41 (127.0.98.2:40451)
W20260620 14:19:22.734901 5332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:22.735141 5332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:22.735180 5332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:22.736475 5332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:22.736539 5332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:22.736583 5332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:22.737958 5332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:22.738225 5332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:22.738526 5332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:22.743582 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.743580 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.743618 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:22.744050 5332 server_base.cc:1034] running on GCE node
I20260620 14:19:22.744266 5332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:22.744464 5332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:22.745604 5332 hybrid_clock.cc:648] HybridClock initialized: now 1781965162745592 us; error 37 us; skew 500 ppm
I20260620 14:19:22.746944 5332 webserver.cc:460] Webserver started at http://127.0.98.3:46859/ using document root <none> and password file <none>
I20260620 14:19:22.747202 5332 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:22.747288 5332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:22.747454 5332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:22.748577 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "91a64c3a0e4e40c0b164e624473570de"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "8116d1a20d8c753255761e631a014961"
server_key_iv: "0437de1ba0878fd18f59289801f49341"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.748925 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "91a64c3a0e4e40c0b164e624473570de"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "8116d1a20d8c753255761e631a014961"
server_key_iv: "0437de1ba0878fd18f59289801f49341"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.750258 5332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:22.750993 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.751185 5332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:22.751289 5332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "91a64c3a0e4e40c0b164e624473570de"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "8116d1a20d8c753255761e631a014961"
server_key_iv: "0437de1ba0878fd18f59289801f49341"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.751375 5332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:22.754686 5332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:22.754951 5332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:22.755093 5332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:22.755288 5332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:22.755635 5332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:22.755695 5332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.755734 5332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:22.755762 5332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.761597 5332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33711
I20260620 14:19:22.761639 5459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33711 every 8 connection(s)
I20260620 14:19:22.761950 5332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:22.765262 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.762629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53823 (local address 127.0.98.62:39401)
0620 14:19:22.762694 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.762696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.763205 (+ 509us) server_negotiation.cc:407] Connection header received
0620 14:19:22.763797 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.763799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.763805 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.763862 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.765001 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.765003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.765003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.765008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.765024 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.765029 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.765092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.765169 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.765185 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:19:22.765446 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.762620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39401 (local address 127.0.98.3:53823)
0620 14:19:22.762833 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.763090 (+ 257us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.763685 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.763934 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.763938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.763948 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.764848 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.764849 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.764877 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.764904 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.765090 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.765095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.765098 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.765174 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.765236 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
I20260620 14:19:22.765944 5460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39401
I20260620 14:19:22.766036 5460 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:22.766149 5460 heartbeater.cc:511] Master 127.0.98.62:39401 requested a full tablet report, sending...
I20260620 14:19:22.766474 5010 ts_manager.cc:194] Registered new tserver with Master: 91a64c3a0e4e40c0b164e624473570de (127.0.98.3:33711)
I20260620 14:19:22.767256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5332
I20260620 14:19:22.767335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:22.767479 392 external_mini_cluster.cc:1428] Setting key ab3cfb8827a65f187f5c3449302b634b
I20260620 14:19:22.768342 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:19:22.770736 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.769604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.0.98.62:39401)
0620 14:19:22.769666 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.769668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.769768 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:19:22.769846 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.769846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.769851 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.769908 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.770514 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.770515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.770516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.770520 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.770534 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.770538 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.770593 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.770649 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.770666 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:22.772312 5010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:22.773834 5010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:22.777748 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.775887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40451 (local address 127.0.0.1:46280)
0620 14:19:22.776132 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.776176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.776234 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.776758 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.776764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.776770 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.777278 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.777279 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.777296 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.777322 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.777587 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.777594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.777597 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.777666 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.777681 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260620 14:19:22.777812 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.775974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.0.98.2:40451)
0620 14:19:22.776304 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.776309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.776320 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:22.776349 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.776351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.776361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.776431 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.777402 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.777415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.777416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.777421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.777459 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.777487 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.777547 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.777685 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.777704 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260620 14:19:22.778038 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.775915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37141 (local address 127.0.0.1:39698)
0620 14:19:22.776475 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.776484 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.776524 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.777022 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.777023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.777025 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.777612 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.777613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.777618 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.777628 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.777905 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.777905 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.777906 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.777969 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.777982 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":75,"threads_started":1}
I20260620 14:19:22.778080 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.776134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.0.98.1:37141)
0620 14:19:22.776395 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.776398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.776631 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:19:22.776660 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.776663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.776673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.776747 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.777707 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.777715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.777715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.777719 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.777754 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.777780 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.777860 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.777981 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.777997 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
I20260620 14:19:22.778481 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.775897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33711 (local address 127.0.0.1:53596)
0620 14:19:22.776320 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.776329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.776375 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.776839 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.776840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.776843 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.777454 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.777455 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.777461 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.777472 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.777723 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.777724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.777724 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.777769 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.777782 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":64,"threads_started":1}
I20260620 14:19:22.778621 5263 tablet_service.cc:1467] Processing CreateTablet for tablet abdcf138ff70495893877cc0cf958a2f (DEFAULT_TABLE table=test-table [id=c8fd2bf045824def9a6f30f1f5cb76c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:22.778756 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.776128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.98.3:33711)
0620 14:19:22.776304 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.776308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.776389 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:19:22.776440 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.776442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.776450 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.776550 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.777561 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.777567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.777568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.777573 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.777608 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.777631 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.777695 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.778640 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.778656 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260620 14:19:22.778759 5132 tablet_service.cc:1467] Processing CreateTablet for tablet abdcf138ff70495893877cc0cf958a2f (DEFAULT_TABLE table=test-table [id=c8fd2bf045824def9a6f30f1f5cb76c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:22.779024 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdcf138ff70495893877cc0cf958a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:22.779086 5132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdcf138ff70495893877cc0cf958a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:22.779994 5394 tablet_service.cc:1467] Processing CreateTablet for tablet abdcf138ff70495893877cc0cf958a2f (DEFAULT_TABLE table=test-table [id=c8fd2bf045824def9a6f30f1f5cb76c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:22.780303 5394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdcf138ff70495893877cc0cf958a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:22.781677 5474 tablet_bootstrap.cc:492] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Bootstrap starting.
I20260620 14:19:22.782217 5475 tablet_bootstrap.cc:492] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2: Bootstrap starting.
I20260620 14:19:22.782547 5474 tablet_bootstrap.cc:654] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:22.782867 5475 tablet_bootstrap.cc:654] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:22.782891 5474 log.cc:824] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:22.783179 5475 log.cc:824] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:22.783738 5474 tablet_bootstrap.cc:492] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: No bootstrap required, opened a new log
I20260620 14:19:22.783809 5474 ts_tablet_manager.cc:1397] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:22.783954 5475 tablet_bootstrap.cc:492] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2: No bootstrap required, opened a new log
I20260620 14:19:22.784034 5475 ts_tablet_manager.cc:1397] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:22.785642 5474 raft_consensus.cc:348] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.785808 5474 raft_consensus.cc:374] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:22.785728 5475 raft_consensus.cc:348] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.785851 5474 raft_consensus.cc:729] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3667c79e1ab542548327bc722bd4ab41, State: Initialized, Role: FOLLOWER
I20260620 14:19:22.785883 5475 raft_consensus.cc:374] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:22.785928 5475 raft_consensus.cc:729] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26045cb5754407080414b7a0a145bf2, State: Initialized, Role: FOLLOWER
I20260620 14:19:22.785980 5474 consensus_queue.cc:260] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.786029 5475 consensus_queue.cc:260] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.786243 5474 ts_tablet_manager.cc:1428] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:19:22.786268 5329 heartbeater.cc:503] Master 127.0.98.62:39401 was elected leader, sending a full tablet report...
I20260620 14:19:22.786290 5475 ts_tablet_manager.cc:1428] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:19:22.786304 5198 heartbeater.cc:503] Master 127.0.98.62:39401 was elected leader, sending a full tablet report...
I20260620 14:19:22.786331 5477 tablet_bootstrap.cc:492] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Bootstrap starting.
I20260620 14:19:22.787222 5477 tablet_bootstrap.cc:654] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:22.787575 5477 log.cc:824] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:22.788195 5477 tablet_bootstrap.cc:492] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: No bootstrap required, opened a new log
I20260620 14:19:22.788275 5477 ts_tablet_manager.cc:1397] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 14:19:22.790112 5477 raft_consensus.cc:348] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.790257 5477 raft_consensus.cc:374] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:22.790304 5477 raft_consensus.cc:729] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a64c3a0e4e40c0b164e624473570de, State: Initialized, Role: FOLLOWER
I20260620 14:19:22.790423 5477 consensus_queue.cc:260] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.790644 5477 ts_tablet_manager.cc:1428] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:19:22.790693 5460 heartbeater.cc:503] Master 127.0.98.62:39401 was elected leader, sending a full tablet report...
I20260620 14:19:22.830770 5480 raft_consensus.cc:482] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:22.830919 5480 raft_consensus.cc:504] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.831265 5480 leader_election.cc:288] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3667c79e1ab542548327bc722bd4ab41 (127.0.98.2:40451), 91a64c3a0e4e40c0b164e624473570de (127.0.98.3:33711)
I20260620 14:19:22.832576 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.831459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40451 (local address 127.0.98.1:39303)
0620 14:19:22.831517 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.831529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.831585 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.831721 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.831721 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.831724 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.832156 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.832156 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.832163 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.832195 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.832412 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.832413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.832414 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.832484 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.832498 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:19:22.832588 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.831573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50593 (local address 127.0.98.3:33711)
0620 14:19:22.831638 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.831640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.831764 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:19:22.831840 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.831841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.831847 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.831934 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.832350 (+ 416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.832351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.832351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.832356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.832370 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.832375 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.832451 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.832514 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.832527 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:22.832588 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.831503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39303 (local address 127.0.98.2:40451)
0620 14:19:22.831571 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.831574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.831584 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:22.831654 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.831655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.831662 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.831717 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.832290 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.832292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.832292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.832297 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.832334 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.832340 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.832403 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.832499 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.832514 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:19:22.832587 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.831475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33711 (local address 127.0.98.1:50593)
0620 14:19:22.831697 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.831705 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:22.831775 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:22.831937 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:22.831938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:22.831940 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:22.832257 (+ 317us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:22.832258 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:22.832263 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:22.832273 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:22.832452 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:22.832453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:22.832453 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:19:22.832503 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:19:22.832513 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":150,"threads_started":1}
I20260620 14:19:22.832933 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdcf138ff70495893877cc0cf958a2f" candidate_uuid: "c26045cb5754407080414b7a0a145bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3667c79e1ab542548327bc722bd4ab41" is_pre_election: true
I20260620 14:19:22.833055 5283 raft_consensus.cc:2454] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c26045cb5754407080414b7a0a145bf2 in term 0.
I20260620 14:19:22.833267 5086 leader_election.cc:302] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3667c79e1ab542548327bc722bd4ab41, c26045cb5754407080414b7a0a145bf2; no voters:
I20260620 14:19:22.833249 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdcf138ff70495893877cc0cf958a2f" candidate_uuid: "c26045cb5754407080414b7a0a145bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a64c3a0e4e40c0b164e624473570de" is_pre_election: true
I20260620 14:19:22.833385 5480 raft_consensus.cc:2781] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:22.833400 5414 raft_consensus.cc:2454] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c26045cb5754407080414b7a0a145bf2 in term 0.
I20260620 14:19:22.833441 5480 raft_consensus.cc:482] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:22.833465 5480 raft_consensus.cc:3037] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:22.834192 5480 raft_consensus.cc:504] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.834342 5480 leader_election.cc:288] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [CANDIDATE]: Term 1 election: Requested vote from peers 3667c79e1ab542548327bc722bd4ab41 (127.0.98.2:40451), 91a64c3a0e4e40c0b164e624473570de (127.0.98.3:33711)
I20260620 14:19:22.834491 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdcf138ff70495893877cc0cf958a2f" candidate_uuid: "c26045cb5754407080414b7a0a145bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3667c79e1ab542548327bc722bd4ab41"
I20260620 14:19:22.834508 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdcf138ff70495893877cc0cf958a2f" candidate_uuid: "c26045cb5754407080414b7a0a145bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a64c3a0e4e40c0b164e624473570de"
I20260620 14:19:22.834569 5283 raft_consensus.cc:3037] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:22.834576 5414 raft_consensus.cc:3037] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:22.835430 5283 raft_consensus.cc:2454] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c26045cb5754407080414b7a0a145bf2 in term 1.
I20260620 14:19:22.835430 5414 raft_consensus.cc:2454] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c26045cb5754407080414b7a0a145bf2 in term 1.
I20260620 14:19:22.835618 5087 leader_election.cc:302] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a64c3a0e4e40c0b164e624473570de, c26045cb5754407080414b7a0a145bf2; no voters:
I20260620 14:19:22.835722 5480 raft_consensus.cc:2781] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:22.835916 5480 raft_consensus.cc:686] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [term 1 LEADER]: Becoming Leader. State: Replica: c26045cb5754407080414b7a0a145bf2, State: Running, Role: LEADER
I20260620 14:19:22.836009 5480 consensus_queue.cc:237] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } }
I20260620 14:19:22.836880 5009 catalog_manager.cc:5495] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c26045cb5754407080414b7a0a145bf2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c26045cb5754407080414b7a0a145bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c26045cb5754407080414b7a0a145bf2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37141 } health_report { overall_health: HEALTHY } } }
I20260620 14:19:22.840776 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.839872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.0.98.1:37141)
0620 14:19:22.839924 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:22.839926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:22.839935 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:22.840007 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:22.840008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:22.840019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:22.840076 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:22.840559 (+ 483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:22.840560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:22.840560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:22.840565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:22.840578 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:22.840582 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:22.840639 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:22.840694 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:19:22.840707 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:19:22.842466 5414 raft_consensus.cc:1261] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 1 FOLLOWER]: Refusing update from remote peer c26045cb5754407080414b7a0a145bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:22.842442 5283 raft_consensus.cc:1261] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 1 FOLLOWER]: Refusing update from remote peer c26045cb5754407080414b7a0a145bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:22.842813 5480 consensus_queue.cc:1035] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a64c3a0e4e40c0b164e624473570de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:22.842947 5484 consensus_queue.cc:1035] T abdcf138ff70495893877cc0cf958a2f P c26045cb5754407080414b7a0a145bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3667c79e1ab542548327bc722bd4ab41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:22.845378 5488 mvcc.cc:205] Tried to move back new op lower bound from 7298929307000115200 to 7298929306977013760. Current Snapshot: MvccSnapshot[applied={T|T < 7298929307000115200}]
I20260620 14:19:22.845988 5490 mvcc.cc:205] Tried to move back new op lower bound from 7298929307000115200 to 7298929306977013760. Current Snapshot: MvccSnapshot[applied={T|T < 7298929307000115200}]
I20260620 14:19:22.846407 5487 mvcc.cc:205] Tried to move back new op lower bound from 7298929307000115200 to 7298929306977013760. Current Snapshot: MvccSnapshot[applied={T|T < 7298929307000115200 or (T in {7298929307000115200})}]
I20260620 14:19:22.847234 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:22.847329 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:22.848049 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46940e70a48e499e8873dc04265c4117: Sending DeleteTablet for 3 replicas of tablet abdcf138ff70495893877cc0cf958a2f
I20260620 14:19:22.848393 5394 tablet_service.cc:1514] Processing DeleteTablet for tablet abdcf138ff70495893877cc0cf958a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:22 UTC) from {username='slave'} at 127.0.0.1:53596
I20260620 14:19:22.848455 5263 tablet_service.cc:1514] Processing DeleteTablet for tablet abdcf138ff70495893877cc0cf958a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:22 UTC) from {username='slave'} at 127.0.0.1:46280
I20260620 14:19:22.848661 5498 tablet_replica.cc:330] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: stopping tablet replica
I20260620 14:19:22.848667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5070
I20260620 14:19:22.848652 5132 tablet_service.cc:1514] Processing DeleteTablet for tablet abdcf138ff70495893877cc0cf958a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:22 UTC) from {username='slave'} at 127.0.0.1:39698
I20260620 14:19:22.848825 5498 raft_consensus.cc:2229] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:22.849623 5498 raft_consensus.cc:2258] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:22.850042 5498 ts_tablet_manager.cc:1905] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:22.851701 5497 tablet_replica.cc:330] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: stopping tablet replica
I20260620 14:19:22.851809 5497 raft_consensus.cc:2229] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:22.851960 5497 raft_consensus.cc:2258] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:22.851970 5498 ts_tablet_manager.cc:1918] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:22.852034 5498 log.cc:1196] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/abdcf138ff70495893877cc0cf958a2f
I20260620 14:19:22.852284 5498 ts_tablet_manager.cc:1939] T abdcf138ff70495893877cc0cf958a2f P 3667c79e1ab542548327bc722bd4ab41: Deleting consensus metadata
I20260620 14:19:22.852403 5497 ts_tablet_manager.cc:1905] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:22.852701 4995 catalog_manager.cc:4841] TS 3667c79e1ab542548327bc722bd4ab41 (127.0.98.2:40451): tablet abdcf138ff70495893877cc0cf958a2f (table test-table [id=c8fd2bf045824def9a6f30f1f5cb76c9]) successfully deleted
I20260620 14:19:22.853876 5497 ts_tablet_manager.cc:1918] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:22.853955 5497 log.cc:1196] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/abdcf138ff70495893877cc0cf958a2f
I20260620 14:19:22.854183 5497 ts_tablet_manager.cc:1939] T abdcf138ff70495893877cc0cf958a2f P 91a64c3a0e4e40c0b164e624473570de: Deleting consensus metadata
I20260620 14:19:22.854600 4996 catalog_manager.cc:4841] TS 91a64c3a0e4e40c0b164e624473570de (127.0.98.3:33711): tablet abdcf138ff70495893877cc0cf958a2f (table test-table [id=c8fd2bf045824def9a6f30f1f5cb76c9]) successfully deleted
W20260620 14:19:22.854691 4994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37141 (error 108)
I20260620 14:19:22.854940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5201
I20260620 14:19:22.855504 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:22.855054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37141 (local address 127.0.0.1:39726)
0620 14:19:22.855427 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:22.855465 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
W20260620 14:19:22.857113 4994 catalog_manager.cc:4584] TS c26045cb5754407080414b7a0a145bf2 (127.0.98.1:37141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abdcf138ff70495893877cc0cf958a2f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37141: connect: Connection refused (error 111)
I20260620 14:19:22.860242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5332
I20260620 14:19:22.865288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 4979
2026-06-20T14:19:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (649 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:19:22.884358 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:19:22.885398 392 webserver.cc:460] Webserver started at https://127.0.0.1:36765/ using document root <none> and password file <none>
I20260620 14:19:22.885514 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:19:22.885689 392 webserver.cc:460] Webserver started at http://127.0.0.1:36921/ using document root <none> and password file <none>
2026-06-20T14:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:22Z Disabled control of system clock
I20260620 14:19:22.893473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33101
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33101
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36765/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/testchainca.cert with env {}
W20260620 14:19:22.972930 5505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:22.973130 5505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:22.973151 5505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:22.974239 5505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:22.974272 5505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:22.974284 5505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:22.974298 5505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:22.974309 5505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:19:22.974323 5505 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:19:22.974334 5505 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36765/jwks.json
I20260620 14:19:22.975768 5505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36765/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:22.975960 5505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:22.976116 5505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:22.980917 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.980926 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:22.980926 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:22.981066 5505 server_base.cc:1034] running on GCE node
I20260620 14:19:22.981355 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:22.981650 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:22.982815 5505 hybrid_clock.cc:648] HybridClock initialized: now 1781965162982809 us; error 37 us; skew 500 ppm
I20260620 14:19:22.984120 5505 webserver.cc:460] Webserver started at http://127.0.98.62:43399/ using document root <none> and password file <none>
I20260620 14:19:22.984359 5505 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:22.984443 5505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:22.984596 5505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:22.985663 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "35ea424f57b24540937905f2c7095696"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "f57365a732e5defc7fbec6d62da09e32"
server_key_iv: "afe455a1a5de539f9ccc0c3eb0f91f17"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.986022 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "35ea424f57b24540937905f2c7095696"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "f57365a732e5defc7fbec6d62da09e32"
server_key_iv: "afe455a1a5de539f9ccc0c3eb0f91f17"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.987386 5505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:19:22.988204 5519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:22.988372 5505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:22.988483 5505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "35ea424f57b24540937905f2c7095696"
format_stamp: "Formatted at 2026-06-20 14:19:22 on dist-test-slave-h6n5"
server_key: "f57365a732e5defc7fbec6d62da09e32"
server_key_iv: "afe455a1a5de539f9ccc0c3eb0f91f17"
server_key_version: "encryptionkey@0"
I20260620 14:19:22.988571 5505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:22.998427 5505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:22.998754 5505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:22.998914 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:23.002808 5505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33101
I20260620 14:19:23.002843 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33101 every 8 connection(s)
I20260620 14:19:23.003185 5505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:23.004204 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:23.006431 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696: Bootstrap starting.
I20260620 14:19:23.006973 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:23.007213 5572 log.cc:824] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:23.007695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5505
I20260620 14:19:23.007804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:23.008004 392 external_mini_cluster.cc:1428] Setting key df594f8d18cff4d65594ecfc078ab418
I20260620 14:19:23.008034 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696: No bootstrap required, opened a new log
I20260620 14:19:23.009905 5572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ea424f57b24540937905f2c7095696" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } }
I20260620 14:19:23.010042 5572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:23.010064 5572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ea424f57b24540937905f2c7095696, State: Initialized, Role: FOLLOWER
I20260620 14:19:23.010233 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ea424f57b24540937905f2c7095696" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } }
I20260620 14:19:23.010324 5572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:23.010362 5572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:23.010421 5572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:23.010982 5572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ea424f57b24540937905f2c7095696" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } }
I20260620 14:19:23.011108 5572 leader_election.cc:302] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35ea424f57b24540937905f2c7095696; no voters:
I20260620 14:19:23.011298 5572 leader_election.cc:288] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:23.011327 5577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:23.011435 5577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [term 1 LEADER]: Becoming Leader. State: Replica: 35ea424f57b24540937905f2c7095696, State: Running, Role: LEADER
I20260620 14:19:23.011528 5577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ea424f57b24540937905f2c7095696" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } }
I20260620 14:19:23.011590 5572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:23.011859 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35ea424f57b24540937905f2c7095696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ea424f57b24540937905f2c7095696" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } }
I20260620 14:19:23.012105 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:23.012028 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35ea424f57b24540937905f2c7095696. Latest consensus state: current_term: 1 leader_uuid: "35ea424f57b24540937905f2c7095696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ea424f57b24540937905f2c7095696" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } }
I20260620 14:19:23.012545 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:23.012713 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:23.013234 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:23.015069 5586 catalog_manager.cc:1347] Generated new cluster ID: c18d0151b2af46b5b091e10467a29b0d
I20260620 14:19:23.015120 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:23.015877 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.008571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.0.98.62:33101)
0620 14:19:23.008774 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.008784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.008809 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:19:23.009893 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.009903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.009968 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.010161 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:23.010402 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.011971 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.013133 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.013764 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.013908 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.014605 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:23.014608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:23.014609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:23.014814 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:23.014856 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:23.014872 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:23.014949 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:23.015644 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 14:19:23.015729 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 14:19:23.020037 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:19:23.026787 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:23.027336 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:23.033499 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35ea424f57b24540937905f2c7095696: Generated new TSK 0
I20260620 14:19:23.033681 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:19:23.056715 5602 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:19:23.056859 5602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:19:23.057077 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.041662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.98.62:33101)
0620 14:19:23.041724 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.041726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.041890 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:19:23.041952 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.041953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.041966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.042047 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:19:23.042180 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.042953 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.043746 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.044054 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.044125 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.044310 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:19:23.056865 (+ 12555us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:19:23.057006 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:19:23.057137 5576 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:19:23.057449 5601 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:23.041612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33101 (local address 127.0.0.1:40206)
0620 14:19:23.041814 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:23.041823 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:23.041878 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:23.042048 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:23.042048 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:23.042050 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:19:23.042113 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.042114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.043035 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.043035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.043668 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.043671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.044129 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.044131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.044178 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.044196 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:19:23.057012 (+ 12816us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:19:23.057109 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:19:23.057904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5505
2026-06-20T14:19:23Z chronyd exiting
I20260620 14:19:23.093880 392 test_util.cc:175] -----------------------------------------------
I20260620 14:19:23.093967 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965139717380-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:19:23.094679 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:19:23.095327 392 webserver.cc:460] Webserver started at https://127.0.0.1:34659/ using document root <none> and password file <none>
I20260620 14:19:23.095369 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:19:23.095534 392 webserver.cc:460] Webserver started at http://127.0.0.1:46571/ using document root <none> and password file <none>
2026-06-20T14:19:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:23Z Disabled control of system clock
I20260620 14:19:23.103281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34659/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/testchainca.cert with env {}
W20260620 14:19:23.182528 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:23.182734 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:23.182756 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:23.183936 5608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:23.183976 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:23.183990 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:23.184002 5608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:23.184015 5608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:19:23.184026 5608 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:19:23.184047 5608 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34659/jwks.json
I20260620 14:19:23.185529 5608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34659/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:23.185721 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:23.185880 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:23.190641 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:23.190665 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:23.190769 5608 server_base.cc:1034] running on GCE node
W20260620 14:19:23.190944 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:23.191215 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:23.191524 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:23.192744 5608 hybrid_clock.cc:648] HybridClock initialized: now 1781965163192724 us; error 40 us; skew 500 ppm
I20260620 14:19:23.194049 5608 webserver.cc:460] Webserver started at http://127.0.98.62:33659/ using document root <none> and password file <none>
I20260620 14:19:23.194289 5608 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:23.194357 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:23.194519 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:23.195645 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "050a9ad4294e41a5b50e5bcd45ceb48f"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "fab6fa7dafcd4cc5ef23241b56c633c7"
server_key_iv: "d40ab2adef9c358e9957f1b4f0054a1b"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.196013 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "050a9ad4294e41a5b50e5bcd45ceb48f"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "fab6fa7dafcd4cc5ef23241b56c633c7"
server_key_iv: "d40ab2adef9c358e9957f1b4f0054a1b"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.197378 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:19:23.198135 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:23.198302 5608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:23.198400 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "050a9ad4294e41a5b50e5bcd45ceb48f"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "fab6fa7dafcd4cc5ef23241b56c633c7"
server_key_iv: "d40ab2adef9c358e9957f1b4f0054a1b"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.198490 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:23.206848 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:23.207146 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:23.207289 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:23.210973 5608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42743
I20260620 14:19:23.211004 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42743 every 8 connection(s)
I20260620 14:19:23.211325 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:23.212318 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:23.214484 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f: Bootstrap starting.
I20260620 14:19:23.215051 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:23.215332 5675 log.cc:824] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:23.216076 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f: No bootstrap required, opened a new log
I20260620 14:19:23.217118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5608
I20260620 14:19:23.217240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:23.217451 392 external_mini_cluster.cc:1428] Setting key d09cd05785e766efc5090e317cec19ed
I20260620 14:19:23.217914 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260620 14:19:23.218106 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:23.218146 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050a9ad4294e41a5b50e5bcd45ceb48f, State: Initialized, Role: FOLLOWER
I20260620 14:19:23.218274 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260620 14:19:23.218348 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:23.218377 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:23.218431 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:23.219172 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260620 14:19:23.219281 5675 leader_election.cc:302] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050a9ad4294e41a5b50e5bcd45ceb48f; no voters:
I20260620 14:19:23.219426 5675 leader_election.cc:288] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:23.219489 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:23.219691 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:23.219715 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [term 1 LEADER]: Becoming Leader. State: Replica: 050a9ad4294e41a5b50e5bcd45ceb48f, State: Running, Role: LEADER
I20260620 14:19:23.219800 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260620 14:19:23.220135 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 050a9ad4294e41a5b50e5bcd45ceb48f. Latest consensus state: current_term: 1 leader_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260620 14:19:23.220232 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:23.220147 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050a9ad4294e41a5b50e5bcd45ceb48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260620 14:19:23.220315 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f [sys.catalog]: This master's current role is: LEADER
W20260620 14:19:23.221340 5695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:19:23.221498 5695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:19:23.221561 5689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:23.221715 5689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:23.223489 5689 catalog_manager.cc:1347] Generated new cluster ID: 66464d720bac49e9a4002c013d387f10
I20260620 14:19:23.223541 5689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:23.223845 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.217968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.0.98.62:42743)
0620 14:19:23.218197 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.218206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.218237 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:19:23.218349 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.218356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.218444 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.218594 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:23.218746 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.220599 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.221677 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.222166 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.222294 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.223037 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:23.223040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:23.223041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:23.223267 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:23.223307 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:23.223321 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:23.223392 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:23.223612 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:19:23.223690 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260620 14:19:23.224709 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:19:23.245173 5689 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:23.245735 5689 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:23.255791 5689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 050a9ad4294e41a5b50e5bcd45ceb48f: Generated new TSK 0
I20260620 14:19:23.256029 5689 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:19:23.286985 5705 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:19:23.287076 5705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:19:23.287393 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.264808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.0.98.62:42743)
0620 14:19:23.264874 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.264877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.264998 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:19:23.265078 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.265078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.265090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.265150 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:19:23.265313 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.265825 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.267113 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.267322 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.267386 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.267643 (+ 257us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:19:23.287148 (+ 19505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:19:23.287339 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 14:19:23.287469 5679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:19:23.287520 5704 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:23.264749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.0.1:53592)
0620 14:19:23.264922 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:23.264934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:23.265005 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:23.265181 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:23.265182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:23.265184 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:19:23.265245 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.265248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.265925 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.265926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.267001 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.267005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.267412 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.267416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.267477 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.267483 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:19:23.287300 (+ 19817us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:19:23.287420 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260620 14:19:23.287945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5608
2026-06-20T14:19:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:19:23.327059 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:19:23.327936 392 webserver.cc:460] Webserver started at https://127.0.0.1:44709/ using document root <none> and password file <none>
I20260620 14:19:23.327999 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:19:23.328154 392 webserver.cc:460] Webserver started at http://127.0.0.1:44727/ using document root <none> and password file <none>
2026-06-20T14:19:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:23Z Disabled control of system clock
I20260620 14:19:23.335721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40733
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40733
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44709/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:19:23.413662 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:23.413861 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:23.413882 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:23.414978 5711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:23.415012 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:23.415025 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:23.415037 5711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:23.415050 5711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:19:23.415061 5711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:19:23.415088 5711 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44709/jwks.json
I20260620 14:19:23.416396 5711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44709/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:23.416581 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:23.416808 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:23.421727 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:23.421826 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:23.421809 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:23.421923 5711 server_base.cc:1034] running on GCE node
I20260620 14:19:23.422173 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:23.422458 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:23.423632 5711 hybrid_clock.cc:648] HybridClock initialized: now 1781965163423614 us; error 42 us; skew 500 ppm
I20260620 14:19:23.424882 5711 webserver.cc:460] Webserver started at http://127.0.98.62:42909/ using document root <none> and password file <none>
I20260620 14:19:23.425143 5711 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:23.425213 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:23.425379 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:23.426450 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "9059222e2d1e4c1caf1623761889c217"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "5d48f538af5db9a7945e98efe5895ee3"
server_key_iv: "87690a95a1e19d8c5362742430387ae7"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.427000 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "9059222e2d1e4c1caf1623761889c217"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "5d48f538af5db9a7945e98efe5895ee3"
server_key_iv: "87690a95a1e19d8c5362742430387ae7"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.428517 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:23.429422 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:23.429626 5711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:23.429733 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "9059222e2d1e4c1caf1623761889c217"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "5d48f538af5db9a7945e98efe5895ee3"
server_key_iv: "87690a95a1e19d8c5362742430387ae7"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.429828 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:23.447217 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:23.447702 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:23.447883 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:23.451982 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40733
I20260620 14:19:23.452055 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40733 every 8 connection(s)
I20260620 14:19:23.452476 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:23.453334 5778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:23.455977 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217: Bootstrap starting.
I20260620 14:19:23.456511 5778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:23.456748 5778 log.cc:824] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:23.457438 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217: No bootstrap required, opened a new log
I20260620 14:19:23.459043 5778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9059222e2d1e4c1caf1623761889c217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40733 } }
I20260620 14:19:23.459182 5778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:23.459205 5778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9059222e2d1e4c1caf1623761889c217, State: Initialized, Role: FOLLOWER
I20260620 14:19:23.459309 5778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9059222e2d1e4c1caf1623761889c217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40733 } }
I20260620 14:19:23.459362 5778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:23.459383 5778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:23.459407 5778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:23.459671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5711
I20260620 14:19:23.459767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:23.459942 392 external_mini_cluster.cc:1428] Setting key 7762df128577938dbe74b2c5cfa374c9
I20260620 14:19:23.460062 5778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9059222e2d1e4c1caf1623761889c217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40733 } }
I20260620 14:19:23.460193 5778 leader_election.cc:302] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9059222e2d1e4c1caf1623761889c217; no voters:
I20260620 14:19:23.460390 5778 leader_election.cc:288] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:23.460505 5781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:23.460652 5781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [term 1 LEADER]: Becoming Leader. State: Replica: 9059222e2d1e4c1caf1623761889c217, State: Running, Role: LEADER
I20260620 14:19:23.460697 5778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:23.460880 5781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9059222e2d1e4c1caf1623761889c217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40733 } }
I20260620 14:19:23.461390 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9059222e2d1e4c1caf1623761889c217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9059222e2d1e4c1caf1623761889c217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40733 } } }
I20260620 14:19:23.461488 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:23.461616 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9059222e2d1e4c1caf1623761889c217. Latest consensus state: current_term: 1 leader_uuid: "9059222e2d1e4c1caf1623761889c217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9059222e2d1e4c1caf1623761889c217" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40733 } } }
I20260620 14:19:23.461669 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:23.462090 5792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:23.462648 5792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:23.464308 5792 catalog_manager.cc:1347] Generated new cluster ID: d46c19a6c7c647e1bf095777772418bf
I20260620 14:19:23.464363 5792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:23.466281 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.460526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.0.98.62:40733)
0620 14:19:23.460872 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.460881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.460911 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:19:23.461030 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.461037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.461106 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.461255 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:23.461589 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.463024 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.464042 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.464593 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.464738 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.465527 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:23.465531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:23.465531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:23.465717 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:23.465758 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:23.465769 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:23.465852 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:23.465945 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:19:23.466119 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260620 14:19:23.467336 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:19:23.471771 5792 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:23.472514 5792 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:23.475222 5792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9059222e2d1e4c1caf1623761889c217: Generated new TSK 0
I20260620 14:19:23.475400 5792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:23.488222 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.484275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.0.98.62:40733)
0620 14:19:23.484323 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.484325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.484410 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:19:23.484481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.484482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.484491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.484546 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:19:23.484667 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.485144 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.486572 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.486757 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.486821 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.487028 (+ 207us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:19:23.487987 (+ 959us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44709/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:19:23.488138 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44709/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 14:19:23.488286 5784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44709/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:19:23.488303 5807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:23.484134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40733 (local address 127.0.0.1:41436)
0620 14:19:23.484337 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:23.484349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:23.484419 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:23.484559 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:23.484559 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:23.484561 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:19:23.484618 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.484620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.485469 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.485470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.486452 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.486456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.486839 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.486847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.486917 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.486923 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:19:23.488106 (+ 1183us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44709/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:19:23.488198 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44709/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":136,"threads_started":1}
I20260620 14:19:23.488593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5711
E20260620 14:19:23.500495 5808 webserver.cc:533] Webserver: sslize failed:
E20260620 14:19:23.500583 5808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:19:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (199 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:19:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:23Z Disabled control of system clock
I20260620 14:19:23.534240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36757
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36757
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/testchainca.cert with env {}
W20260620 14:19:23.613276 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:23.613548 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:23.613590 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:23.614816 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:23.614889 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:23.614912 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:23.614934 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:23.614989 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:19:23.615015 5812 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:19:23.615039 5812 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:19:23.616632 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:23.616923 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:23.617147 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:23.622216 5812 server_base.cc:1034] running on GCE node
W20260620 14:19:23.622171 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:23.622180 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:23.622192 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:23.622594 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:23.622905 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:23.624068 5812 hybrid_clock.cc:648] HybridClock initialized: now 1781965163624052 us; error 28 us; skew 500 ppm
I20260620 14:19:23.625381 5812 webserver.cc:460] Webserver started at http://127.0.98.62:34973/ using document root <none> and password file <none>
I20260620 14:19:23.625614 5812 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:23.625727 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:23.625871 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:23.626995 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "e7980dba3ca94ff78b96e743081d499e"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "b611dcead5b9843c61b1e9344715967d"
server_key_iv: "d2ce0800482dcf59ad48874a68449920"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.627380 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7980dba3ca94ff78b96e743081d499e"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "b611dcead5b9843c61b1e9344715967d"
server_key_iv: "d2ce0800482dcf59ad48874a68449920"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.628829 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:23.629647 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:23.629859 5812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:23.629959 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "e7980dba3ca94ff78b96e743081d499e"
format_stamp: "Formatted at 2026-06-20 14:19:23 on dist-test-slave-h6n5"
server_key: "b611dcead5b9843c61b1e9344715967d"
server_key_iv: "d2ce0800482dcf59ad48874a68449920"
server_key_version: "encryptionkey@0"
I20260620 14:19:23.630041 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:23.644091 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:23.644416 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:23.644565 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:23.648599 5812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36757
I20260620 14:19:23.648639 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36757 every 8 connection(s)
I20260620 14:19:23.648993 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:23.649943 5879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:23.652356 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e: Bootstrap starting.
I20260620 14:19:23.652942 5879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:23.653225 5879 log.cc:824] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:23.653947 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e: No bootstrap required, opened a new log
I20260620 14:19:23.655521 5879 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7980dba3ca94ff78b96e743081d499e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36757 } }
I20260620 14:19:23.655680 5879 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:23.655704 5879 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7980dba3ca94ff78b96e743081d499e, State: Initialized, Role: FOLLOWER
I20260620 14:19:23.655896 5879 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7980dba3ca94ff78b96e743081d499e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36757 } }
I20260620 14:19:23.655980 5879 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:23.656023 5879 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:23.656073 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:23.656631 5879 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7980dba3ca94ff78b96e743081d499e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36757 } }
I20260620 14:19:23.656741 5879 leader_election.cc:302] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7980dba3ca94ff78b96e743081d499e; no voters:
I20260620 14:19:23.656915 5879 leader_election.cc:288] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:23.656984 5882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:23.657152 5882 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [term 1 LEADER]: Becoming Leader. State: Replica: e7980dba3ca94ff78b96e743081d499e, State: Running, Role: LEADER
I20260620 14:19:23.657179 5879 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:23.657279 5882 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7980dba3ca94ff78b96e743081d499e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36757 } }
I20260620 14:19:23.657655 5884 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7980dba3ca94ff78b96e743081d499e. Latest consensus state: current_term: 1 leader_uuid: "e7980dba3ca94ff78b96e743081d499e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7980dba3ca94ff78b96e743081d499e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36757 } } }
I20260620 14:19:23.657763 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7980dba3ca94ff78b96e743081d499e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7980dba3ca94ff78b96e743081d499e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36757 } } }
I20260620 14:19:23.657764 5884 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:23.657872 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:23.658054 5891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:23.658375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5812
I20260620 14:19:23.658492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:23.658555 5891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:23.658715 392 external_mini_cluster.cc:1428] Setting key 9c3bf6c0ff93ae164b9bc31e6d3fbc57
I20260620 14:19:23.660176 5891 catalog_manager.cc:1347] Generated new cluster ID: cb1f6c614289477d8d3cb889966e1595
I20260620 14:19:23.660246 5891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:23.664523 5891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:23.665216 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.659326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.98.62:36757)
0620 14:19:23.659620 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.659628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.659663 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:19:23.659821 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.659828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.659908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.660301 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:23.660392 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.661737 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.662901 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.663596 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.664263 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.664470 (+ 207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:23.664474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:23.664474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:23.664681 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:23.664721 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:23.664735 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:23.664808 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:23.664926 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:19:23.665002 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":67,"threads_started":1}
I20260620 14:19:23.665221 5891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:23.666328 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:19:23.675343 5891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7980dba3ca94ff78b96e743081d499e: Generated new TSK 0
I20260620 14:19:23.675539 5891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:23.722013 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:23.682209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.0.98.62:36757)
0620 14:19:23.682255 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:23.682257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:23.682377 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:19:23.682415 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:23.682416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:23.682428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:23.682512 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:19:23.682638 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.683445 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.684286 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.684611 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.684683 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.684904 (+ 221us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:19:23.721728 (+ 36824us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:19:23.721928 (+ 200us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41108: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 14:19:23.722095 5901 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41108: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:19:23.722115 5908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:23.682140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36757 (local address 127.0.0.1:41108)
0620 14:19:23.682294 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:23.682305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:23.682367 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:23.682517 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:23.682518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:23.682519 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:19:23.682570 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.682572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.683590 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.683591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.684201 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:23.684204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:23.684697 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:23.684702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:23.684793 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:23.684799 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:19:23.721871 (+ 37072us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:19:23.721995 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36757: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":120,"threads_started":1}
I20260620 14:19:23.722571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5812
2026-06-20T14:19:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (212 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:23 dist-test-slave-h6n5 krb5kdc[5920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:23 dist-test-slave-h6n5 krb5kdc[5920](info): set up 2 sockets
Jun 20 14:19:23 dist-test-slave-h6n5 krb5kdc[5920](info): commencing operation
krb5kdc: starting...
W20260620 14:19:24.931835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:25.048388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:25.128975 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:25.129221 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:25.129258 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:25.130417 5933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:25.130487 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:25.130512 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:25.130534 5933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:25.130563 5933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:25.132118 5933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:25.132392 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:25.132611 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:25.137393 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:25.137477 5933 server_base.cc:1034] running on GCE node
W20260620 14:19:25.137452 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:25.137409 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:25.137864 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:25.138101 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:25.139266 5933 hybrid_clock.cc:648] HybridClock initialized: now 1781965165139255 us; error 42 us; skew 500 ppm
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.141686 5933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:25.142153 5933 webserver.cc:460] Webserver started at http://127.0.98.62:39871/ using document root <none> and password file <none>
I20260620 14:19:25.142374 5933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:25.142445 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:25.142577 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:25.143687 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "1a97245781ad45aeb0d1ea891cf85bc4"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "111e36bcace62784be532b428ede4704"
server_key_iv: "d93fe9841ddd325bc7291aa1e565bba7"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.144033 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a97245781ad45aeb0d1ea891cf85bc4"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "111e36bcace62784be532b428ede4704"
server_key_iv: "d93fe9841ddd325bc7291aa1e565bba7"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.145294 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:25.145972 5948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.146131 5933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:25.146224 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "1a97245781ad45aeb0d1ea891cf85bc4"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "111e36bcace62784be532b428ede4704"
server_key_iv: "d93fe9841ddd325bc7291aa1e565bba7"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.146301 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:25.155949 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:25.158164 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:25.158324 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:25.162065 5933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46795
I20260620 14:19:25.162111 6000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46795 every 8 connection(s)
I20260620 14:19:25.162622 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:25.163403 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:25.165652 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4: Bootstrap starting.
I20260620 14:19:25.166185 6001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:25.166425 6001 log.cc:824] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:25.167048 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4: No bootstrap required, opened a new log
I20260620 14:19:25.168643 6001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 14:19:25.168778 6001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:25.168802 6001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a97245781ad45aeb0d1ea891cf85bc4, State: Initialized, Role: FOLLOWER
I20260620 14:19:25.168938 6001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 14:19:25.169027 6001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:25.169050 6001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:25.169077 6001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:25.169907 6001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 14:19:25.169987 6001 leader_election.cc:302] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a97245781ad45aeb0d1ea891cf85bc4; no voters:
I20260620 14:19:25.170106 6001 leader_election.cc:288] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:25.170177 6004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:25.170347 6004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [term 1 LEADER]: Becoming Leader. State: Replica: 1a97245781ad45aeb0d1ea891cf85bc4, State: Running, Role: LEADER
I20260620 14:19:25.170383 6001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:25.170445 6004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 14:19:25.170799 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } } }
I20260620 14:19:25.170804 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a97245781ad45aeb0d1ea891cf85bc4. Latest consensus state: current_term: 1 leader_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a97245781ad45aeb0d1ea891cf85bc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } } }
I20260620 14:19:25.170893 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:25.170907 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:25.171103 6012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:25.171748 6012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:25.172508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 5933
I20260620 14:19:25.172608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:25.172822 392 external_mini_cluster.cc:1428] Setting key 3b341c9686cc0dae94790168a4f46d2e
I20260620 14:19:25.173595 6012 catalog_manager.cc:1347] Generated new cluster ID: 542d536c5ff1407e9dee9b03032de33b
I20260620 14:19:25.173650 6012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:25.183008 6012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:25.183712 6012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:25.184394 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.173286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.0.98.62:46795)
0620 14:19:25.173436 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.173445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.173486 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:19:25.173610 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.173618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.173677 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.173812 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.174610 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.176014 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.177068 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.177687 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.177782 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.180657 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:25.180661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:25.180662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:25.180675 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:25.182696 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.183103 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.183106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.183107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.183209 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.183430 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.183431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.183431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.183722 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:25.183904 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:25.184035 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:19:25.184175 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260620 14:19:25.188055 6012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a97245781ad45aeb0d1ea891cf85bc4: Generated new TSK 0
I20260620 14:19:25.188241 6012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:25.222182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:25.326390 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:25.326581 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:25.326604 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:25.327781 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:25.327819 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:25.327845 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:25.329247 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:25.329476 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:25.329725 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:25.335597 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:25.335632 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:25.335920 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:25.336894 6027 server_base.cc:1034] running on GCE node
I20260620 14:19:25.337121 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:25.337370 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:25.338546 6027 hybrid_clock.cc:648] HybridClock initialized: now 1781965165338532 us; error 27 us; skew 500 ppm
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.341305 6027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:25.341686 6027 webserver.cc:460] Webserver started at http://127.0.98.1:39429/ using document root <none> and password file <none>
I20260620 14:19:25.341885 6027 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:25.341928 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:25.342013 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:25.343032 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "1e6e6ec2f60c4da998aabdcdacb6ff02"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "3f9b3aa36ccac4d7c6119eeeb3826a49"
server_key_iv: "426645803235311cd24ce04af08d14b0"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.343322 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1e6e6ec2f60c4da998aabdcdacb6ff02"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "3f9b3aa36ccac4d7c6119eeeb3826a49"
server_key_iv: "426645803235311cd24ce04af08d14b0"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.344686 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:25.345458 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.345640 6027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:25.345746 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "1e6e6ec2f60c4da998aabdcdacb6ff02"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "3f9b3aa36ccac4d7c6119eeeb3826a49"
server_key_iv: "426645803235311cd24ce04af08d14b0"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.345862 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:25.365942 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:25.368346 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:25.368511 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:25.368760 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:25.369060 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:25.369114 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.369159 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:25.369184 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.375280 6027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45209
I20260620 14:19:25.375321 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45209 every 8 connection(s)
I20260620 14:19:25.375712 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:25.377730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6027
I20260620 14:19:25.377851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:25.378039 392 external_mini_cluster.cc:1428] Setting key 15b1108946e0eefdec3bb4c499a84063
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:25.385965 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.376507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57753 (local address 127.0.98.62:46795)
0620 14:19:25.376574 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.376575 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.376795 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:19:25.377768 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.377769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.377782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.377834 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.378535 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.379059 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.380399 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.380603 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.380664 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.383037 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:25.383039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:25.383039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:25.383046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:25.384522 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.384948 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.384949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.384950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.385013 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.385273 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.385274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.385274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.385401 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:25.385476 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:25.385754 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:19:25.385844 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:19:25.386121 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.376494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.1:57753)
0620 14:19:25.376673 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:25.376708 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:25.377649 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:25.377907 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:25.377912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:25.378241 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:25.378433 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:25.378439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.379157 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.379158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:25.380302 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:25.380305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.380704 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.380706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:25.380855 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.381344 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:25.381354 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:25.382929 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:25.384618 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:25.384625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:25.384627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:25.384852 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:25.385082 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:25.385083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:25.385083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:25.385195 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:25.385477 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:25.385485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:25.385624 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:19:25.385732 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:19:25.385889 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260620 14:19:25.386559 6156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260620 14:19:25.386654 6156 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:25.386790 6156 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260620 14:19:25.387311 5965 ts_manager.cc:194] Registered new tserver with Master: 1e6e6ec2f60c4da998aabdcdacb6ff02 (127.0.98.1:45209)
I20260620 14:19:25.387959 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:25.403002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:25.483428 6161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:25.483698 6161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:25.483738 6161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:25.484926 6161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:25.485002 6161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:25.485047 6161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:25.486533 6161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:25.486797 6161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:25.487032 6161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:25.492040 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:25.492048 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:25.492048 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:25.492606 6161 server_base.cc:1034] running on GCE node
I20260620 14:19:25.492939 6161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:25.493196 6161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:25.494361 6161 hybrid_clock.cc:648] HybridClock initialized: now 1781965165494348 us; error 35 us; skew 500 ppm
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.497169 6161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:25.497673 6161 webserver.cc:460] Webserver started at http://127.0.98.2:46167/ using document root <none> and password file <none>
I20260620 14:19:25.497911 6161 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:25.497958 6161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:25.498129 6161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:25.499207 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "57464c596da242fba5d7d7902a4caf75"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "47600c7c4a5ac817e7fef5427256ef0a"
server_key_iv: "a24147c06c6d9f19f8358359e37b75ba"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.499639 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "57464c596da242fba5d7d7902a4caf75"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "47600c7c4a5ac817e7fef5427256ef0a"
server_key_iv: "a24147c06c6d9f19f8358359e37b75ba"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.500994 6161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:25.501727 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.501902 6161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:25.501999 6161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "57464c596da242fba5d7d7902a4caf75"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "47600c7c4a5ac817e7fef5427256ef0a"
server_key_iv: "a24147c06c6d9f19f8358359e37b75ba"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.502084 6161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:25.511869 6161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:25.514276 6161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:25.514454 6161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:25.514714 6161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:25.515038 6161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:25.515090 6161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.515126 6161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:25.515175 6161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.521292 6161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34781
I20260620 14:19:25.521344 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34781 every 8 connection(s)
I20260620 14:19:25.521667 6161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:25.527694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6161
I20260620 14:19:25.527810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:25.528028 392 external_mini_cluster.cc:1428] Setting key 6d4a26566070e23dcdd4df68587cc520
I20260620 14:19:25.531461 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.522419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46489 (local address 127.0.98.62:46795)
0620 14:19:25.522489 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.522491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.522672 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:19:25.523655 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.523656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.523670 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.523742 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.524310 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.524811 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.525800 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.525978 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.526029 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.528518 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:25.528520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:25.528520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:25.528528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:25.530041 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.530461 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.530462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.530463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.530525 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.530777 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.530778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.530779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.530910 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:25.530993 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:25.531274 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 14:19:25.531357 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:19:25.531695 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.522391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.2:46489)
0620 14:19:25.522540 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:25.522583 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:25.523507 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:25.523797 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:25.523801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:25.524045 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:25.524236 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:25.524240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.524916 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.524918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:25.525723 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:25.525725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.526050 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.526051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:25.526146 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.526734 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:25.526742 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:25.528418 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:25.530142 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:25.530149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:25.530151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:25.530382 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:25.530592 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:25.530593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:25.530593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:25.530710 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:25.530993 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:25.531000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:25.531141 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:19:25.531255 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:19:25.531415 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260620 14:19:25.532159 6290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260620 14:19:25.532251 6290 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:25.532455 6290 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260620 14:19:25.532837 5965 ts_manager.cc:194] Registered new tserver with Master: 57464c596da242fba5d7d7902a4caf75 (127.0.98.2:34781)
I20260620 14:19:25.533390 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46489
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:25.552167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:25.631474 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:25.631693 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:25.631717 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:25.632818 6295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:25.632853 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:25.632879 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:25.634333 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:25.634522 6295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:25.634701 6295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:25.639550 6300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:25.639659 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:25.639577 6301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:25.640064 6295 server_base.cc:1034] running on GCE node
I20260620 14:19:25.640304 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:25.640551 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:25.641686 6295 hybrid_clock.cc:648] HybridClock initialized: now 1781965165641666 us; error 36 us; skew 500 ppm
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.644594 6295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:25.645081 6295 webserver.cc:460] Webserver started at http://127.0.98.3:37677/ using document root <none> and password file <none>
I20260620 14:19:25.645332 6295 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:25.645397 6295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:25.645553 6295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:25.646651 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b5bd6215cb248408e996780b46a1aea"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "7dcf0e14bd155368f2363b0bea7c285e"
server_key_iv: "c533d2e54f8cca023d92f00d827445f2"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.647013 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b5bd6215cb248408e996780b46a1aea"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "7dcf0e14bd155368f2363b0bea7c285e"
server_key_iv: "c533d2e54f8cca023d92f00d827445f2"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.648397 6295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:25.649207 6310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.649386 6295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:25.649462 6295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "4b5bd6215cb248408e996780b46a1aea"
format_stamp: "Formatted at 2026-06-20 14:19:25 on dist-test-slave-h6n5"
server_key: "7dcf0e14bd155368f2363b0bea7c285e"
server_key_iv: "c533d2e54f8cca023d92f00d827445f2"
server_key_version: "encryptionkey@0"
I20260620 14:19:25.649550 6295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:25.657454 6295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:25.659679 6295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:25.659832 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:25.660049 6295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:25.660353 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:25.660384 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.660449 6295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:25.660480 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:25.666146 6295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42867
I20260620 14:19:25.666234 6423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42867 every 8 connection(s)
I20260620 14:19:25.666519 6295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:25.676079 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.667207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41441 (local address 127.0.98.62:46795)
0620 14:19:25.667278 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.667280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.667508 (+ 228us) server_negotiation.cc:407] Connection header received
0620 14:19:25.668514 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.668516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.668529 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.668574 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.669187 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.669697 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.670743 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.671043 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.671123 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.673436 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:25.673438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:25.673438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:25.673446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:25.674686 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.675098 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.675099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.675100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.675156 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.675406 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.675407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.675407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.675525 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:25.675637 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:25.675901 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 14:19:25.675990 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:19:25.676251 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.667209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.3:41441)
0620 14:19:25.667381 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:25.667414 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:25.668385 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:25.668654 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:25.668660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:25.668908 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:25.669092 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:25.669097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.669792 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.669793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:25.670652 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:25.670654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.671127 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.671129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:25.671272 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.671864 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:25.671872 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:25.673329 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:25.674782 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:25.674789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:25.674791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:25.675009 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:25.675229 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:25.675230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:25.675230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:25.675339 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:25.675628 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:25.675634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:25.675769 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:19:25.675879 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:19:25.676027 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260620 14:19:25.676625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6295
I20260620 14:19:25.676719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:25.676791 6424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260620 14:19:25.676887 6424 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:25.676944 392 external_mini_cluster.cc:1428] Setting key 57e5243e973f7942d81c1121c0560274
I20260620 14:19:25.677090 6424 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260620 14:19:25.677490 5965 ts_manager.cc:194] Registered new tserver with Master: 4b5bd6215cb248408e996780b46a1aea (127.0.98.3:42867)
I20260620 14:19:25.677770 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:19:25.677995 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/test-admin.keytab.
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.692257 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:19:25.693893 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:25.693132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.98.1:45209)
0620 14:19:25.693299 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.693304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.693313 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:25.693339 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.693341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.693353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.693424 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.693800 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36386: BlockingRecv error: recv got EOF from 127.0.0.1:36386 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
W20260620 14:19:25.694094 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.695122 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.696153 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:25.703858 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.696722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.0.98.1:45209)
0620 14:19:25.696772 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.696774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.696810 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:19:25.696883 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.696884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.696894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.696956 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.697350 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.698081 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.698793 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.699131 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.699190 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.701333 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:25.701340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:25.701340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:25.701346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:25.702872 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.703175 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.703177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.703178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.703229 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.703390 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.703390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.703391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.703503 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:25.703681 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:25.703718 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:19:25.703781 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:19:25.705706 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:25.705125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.98.1:45209)
0620 14:19:25.705157 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.705159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.705238 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:19:25.705284 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.705284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.705292 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.705336 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.705644 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36400: BlockingRecv error: recv got EOF from 127.0.0.1:36400 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:19:25.705955 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc)
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.706907 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.707770 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:25.714546 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.708271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.98.1:45209)
0620 14:19:25.708329 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.708331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.708357 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:19:25.708418 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.708419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.708427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.708473 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.708791 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.709258 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.709908 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.710098 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.710146 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.712150 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:25.712151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:25.712151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:25.712157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:25.713646 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.713953 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.713955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.713955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.714018 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.714176 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.714176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.714177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.714265 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:25.714334 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:25.714431 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:19:25.714485 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:19:25.716037 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:25.715507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.0.98.1:45209)
0620 14:19:25.715581 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.715583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.715669 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:19:25.715686 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.715686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.715693 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.715730 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.715987 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36420: BlockingRecv error: recv got EOF from 127.0.0.1:36420 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 14:19:25.716091 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965139717380-392-0/krb5kdc/krb5cc)
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.717054 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:25.717948 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:19:25.724537 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:25.718436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.0.98.1:45209)
0620 14:19:25.718488 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:25.718490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:25.718560 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:19:25.718602 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:25.718602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:25.718610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:25.718652 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:25.718987 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.719424 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.720139 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:25.720310 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:25.720360 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:25.722306 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:25.722307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:25.722307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:25.722313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:25.723605 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.723903 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.723904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.723904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.723954 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:25.724100 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:25.724101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:25.724101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:25.724179 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:25.724242 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:25.724334 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:19:25.724392 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:19:25.725291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6027
I20260620 14:19:25.730769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6161
I20260620 14:19:25.735489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6295
I20260620 14:19:25.739846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 5933
2026-06-20T14:19:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1846 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[6444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[6444](info): set up 2 sockets
Jun 20 14:19:25 dist-test-slave-h6n5 krb5kdc[6444](info): commencing operation
krb5kdc: starting...
W20260620 14:19:26.795868 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:26 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:19:26.919340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:27.000552 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:27.000793 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:27.000830 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:27.001989 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:27.002055 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:27.002080 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:27.002102 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:27.002130 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:27.003612 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:27.003880 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:27.004104 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:27.008862 6457 server_base.cc:1034] running on GCE node
W20260620 14:19:27.008774 6463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.008786 6462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.008812 6465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.009347 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:27.009640 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:27.010800 6457 hybrid_clock.cc:648] HybridClock initialized: now 1781965167010776 us; error 38 us; skew 500 ppm
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:27.013244 6457 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:19:27.013741 6457 webserver.cc:460] Webserver started at http://127.0.98.62:36155/ using document root <none> and password file <none>
I20260620 14:19:27.013978 6457 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:27.014047 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:27.014197 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:27.015244 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "10c59e1e1ea54a74ad3fc0f3f761c456"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "d5590effd9012988f1e9b75842a483a9"
server_key_iv: "3a0ae2296be7fec81905fe93fac02547"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.015673 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "10c59e1e1ea54a74ad3fc0f3f761c456"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "d5590effd9012988f1e9b75842a483a9"
server_key_iv: "3a0ae2296be7fec81905fe93fac02547"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.016973 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:27.017669 6472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.017843 6457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:27.017938 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "10c59e1e1ea54a74ad3fc0f3f761c456"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "d5590effd9012988f1e9b75842a483a9"
server_key_iv: "3a0ae2296be7fec81905fe93fac02547"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.018025 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:27.027477 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:27.029306 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:27.029469 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:27.033339 6457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43329
I20260620 14:19:27.033370 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43329 every 8 connection(s)
I20260620 14:19:27.033704 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:27.033946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6457
I20260620 14:19:27.034073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:27.034291 392 external_mini_cluster.cc:1428] Setting key ff7324d5f32b03a2dbc39d72688ea983
I20260620 14:19:27.034837 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:27.037954 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: Bootstrap starting.
I20260620 14:19:27.038596 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:27.038949 6525 log.cc:824] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:27.039727 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: No bootstrap required, opened a new log
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:27.041880 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260620 14:19:27.042057 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:27.042102 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c59e1e1ea54a74ad3fc0f3f761c456, State: Initialized, Role: FOLLOWER
I20260620 14:19:27.042234 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260620 14:19:27.042317 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:27.042346 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:27.042425 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:27.043088 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260620 14:19:27.043174 6525 leader_election.cc:302] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10c59e1e1ea54a74ad3fc0f3f761c456; no voters:
I20260620 14:19:27.043356 6525 leader_election.cc:288] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:27.043414 6530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:27.043630 6530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 1 LEADER]: Becoming Leader. State: Replica: 10c59e1e1ea54a74ad3fc0f3f761c456, State: Running, Role: LEADER
I20260620 14:19:27.043659 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:27.043817 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260620 14:19:27.044209 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c59e1e1ea54a74ad3fc0f3f761c456. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260620 14:19:27.044277 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:27.044245 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260620 14:19:27.044328 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:27.044481 6538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:27.045631 6538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:27.046339 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.034794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.98.62:43329)
0620 14:19:27.035006 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.035013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.035037 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:19:27.035479 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.035486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.035532 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.035705 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.036446 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.037352 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.038392 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.038859 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.038947 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.041645 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.041649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.041649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.041660 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.043752 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.044148 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.044151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.044152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.044262 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.044511 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.044512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.044512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.044816 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.045912 (+ 1096us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.045996 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.046132 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260620 14:19:27.047396 6538 catalog_manager.cc:1347] Generated new cluster ID: eb1a048fae624342b5aa6e9d54fc02fa
I20260620 14:19:27.047439 6538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:27.058359 6538 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:27.058912 6538 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:27.063688 6538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: Generated new TSK 0
I20260620 14:19:27.063864 6538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:19:27.094996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:27.186473 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:27.186693 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:27.186733 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:27.187942 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:27.187999 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:27.188042 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:27.189446 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:27.189702 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:27.189970 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:27.194859 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.194825 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.194890 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.195164 6551 server_base.cc:1034] running on GCE node
I20260620 14:19:27.195389 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:27.195680 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:27.196818 6551 hybrid_clock.cc:648] HybridClock initialized: now 1781965167196803 us; error 33 us; skew 500 ppm
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:27.199296 6551 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:19:27.199811 6551 webserver.cc:460] Webserver started at http://127.0.98.1:35411/ using document root <none> and password file <none>
I20260620 14:19:27.200043 6551 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:27.200088 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:27.200172 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:27.201121 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "acf0c092f8ac435da3112368a2180822"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "49fdefc70af6ec5d6f95510b144872f1"
server_key_iv: "469f5e756ed71b197710248ebba528ad"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.201413 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "acf0c092f8ac435da3112368a2180822"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "49fdefc70af6ec5d6f95510b144872f1"
server_key_iv: "469f5e756ed71b197710248ebba528ad"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.202611 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:27.203224 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.203419 6551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:27.203519 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "acf0c092f8ac435da3112368a2180822"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "49fdefc70af6ec5d6f95510b144872f1"
server_key_iv: "469f5e756ed71b197710248ebba528ad"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.203617 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:27.221572 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:27.223915 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:27.224073 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:27.224313 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:27.224622 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:27.224653 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.224675 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:27.224722 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.230140 6551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35681
I20260620 14:19:27.230176 6679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35681 every 8 connection(s)
I20260620 14:19:27.230513 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:27.239797 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.231243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36195 (local address 127.0.98.62:43329)
0620 14:19:27.231316 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.231319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.231566 (+ 247us) server_negotiation.cc:407] Connection header received
0620 14:19:27.232091 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.232092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.232101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.232152 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.233045 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.233539 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.234507 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.234707 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.234767 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.237174 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.237176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.237176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.237183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.238553 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.238959 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.238960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.238960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.239012 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.239227 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.239227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.239228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.239327 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.239376 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.239659 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.239729 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:27.240017 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.231232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:36195)
0620 14:19:27.231408 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.231443 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.232002 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.232223 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.232226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.232774 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.232969 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.232973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.233612 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.233613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.234416 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.234418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.234804 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.234806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.234922 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.235401 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.235408 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.237070 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.238639 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.238645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.238647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.238868 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.239075 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.239075 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.239075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.239169 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.239380 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.239384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.239505 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.239636 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.239767 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260620 14:19:27.240556 6680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260620 14:19:27.240585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6551
I20260620 14:19:27.240665 6680 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:27.240681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:27.240865 6680 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260620 14:19:27.240880 392 external_mini_cluster.cc:1428] Setting key 63d7c5ed20dcc67745bf7b213e6258db
I20260620 14:19:27.241443 6488 ts_manager.cc:194] Registered new tserver with Master: acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681)
I20260620 14:19:27.242092 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36195
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:19:27.266569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:27.350342 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:27.350533 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:27.350581 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:27.351768 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:27.351809 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:27.351835 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:27.353242 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:27.353477 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:27.353716 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:27.358620 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.358654 6685 server_base.cc:1034] running on GCE node
W20260620 14:19:27.358609 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.358613 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.358983 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:27.359238 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:27.360383 6685 hybrid_clock.cc:648] HybridClock initialized: now 1781965167360365 us; error 35 us; skew 500 ppm
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:27.362923 6685 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:19:27.363379 6685 webserver.cc:460] Webserver started at http://127.0.98.2:38255/ using document root <none> and password file <none>
I20260620 14:19:27.363667 6685 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:27.363715 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:27.363854 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:27.364884 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "a063836e372f4021b8f5f7d65ff18c6b"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "ae65a03ea4c60f04f6fb41eeb601c393"
server_key_iv: "5d788827bad3c2646901289f5c6af755"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.365235 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a063836e372f4021b8f5f7d65ff18c6b"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "ae65a03ea4c60f04f6fb41eeb601c393"
server_key_iv: "5d788827bad3c2646901289f5c6af755"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.366564 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:27.367321 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.367513 6685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:27.367627 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "a063836e372f4021b8f5f7d65ff18c6b"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "ae65a03ea4c60f04f6fb41eeb601c393"
server_key_iv: "5d788827bad3c2646901289f5c6af755"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.367710 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:27.389535 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:27.391856 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:27.392037 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:27.392298 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:27.392628 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:27.392686 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.392776 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:27.392808 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.399222 6685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39859
I20260620 14:19:27.399276 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39859 every 8 connection(s)
I20260620 14:19:27.399683 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:27.401209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6685
I20260620 14:19:27.401326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:27.401522 392 external_mini_cluster.cc:1428] Setting key 844f8a148eec252edcd16bc49c2be9b9
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:27.410816 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.400448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:34501)
0620 14:19:27.400640 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.400678 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.401222 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.401470 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.401474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.402264 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.402500 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.402506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.403247 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.403248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.404731 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.404734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.405301 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.405302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.405454 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.406048 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.406057 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.407578 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.409506 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.409512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.409514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.409725 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.409939 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.409940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.409940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.410031 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.410338 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.410342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.410464 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.410547 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.410661 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260620 14:19:27.410836 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.400483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34501 (local address 127.0.98.62:43329)
0620 14:19:27.400545 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.400547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.400774 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:19:27.401334 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.401335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.401347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.401399 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.402599 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.403142 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.404851 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.405208 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.405299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.407696 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.407698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.407698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.407706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.409424 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.409805 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.409807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.409807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.409868 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.410098 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.410099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.410100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.410265 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.410347 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.410606 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.410702 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:19:27.411585 6814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260620 14:19:27.411698 6814 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:27.411861 6814 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260620 14:19:27.412228 6488 ts_manager.cc:194] Registered new tserver with Master: a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2:39859)
I20260620 14:19:27.412644 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:19:27.429981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:27.511178 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:27.511375 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:27.511399 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:27.512609 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:27.512655 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:27.512681 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:27.514117 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:27.514351 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:27.514580 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:27.520030 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.520013 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.520013 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.520269 6819 server_base.cc:1034] running on GCE node
I20260620 14:19:27.520483 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:27.520750 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:27.521911 6819 hybrid_clock.cc:648] HybridClock initialized: now 1781965167521889 us; error 38 us; skew 500 ppm
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:27.524544 6819 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:19:27.525058 6819 webserver.cc:460] Webserver started at http://127.0.98.3:34685/ using document root <none> and password file <none>
I20260620 14:19:27.525318 6819 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:27.525391 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:27.525538 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:27.526604 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8b0c869b06c4f3fac1d420969f912a6"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "d7a0729b0a25e6ede8a13ee7a5919abe"
server_key_iv: "d91be37597b4a2cc8a66f0fc871a84ba"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.526965 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8b0c869b06c4f3fac1d420969f912a6"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "d7a0729b0a25e6ede8a13ee7a5919abe"
server_key_iv: "d91be37597b4a2cc8a66f0fc871a84ba"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.528442 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:27.529234 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.529452 6819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:27.529556 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "d8b0c869b06c4f3fac1d420969f912a6"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "d7a0729b0a25e6ede8a13ee7a5919abe"
server_key_iv: "d91be37597b4a2cc8a66f0fc871a84ba"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.529639 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:27.539868 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:27.541769 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:27.541934 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:27.542157 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:27.542459 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:27.542513 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.542554 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:27.542582 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.548830 6819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39473
I20260620 14:19:27.548869 6947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39473 every 8 connection(s)
I20260620 14:19:27.549182 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:27.554761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6819
I20260620 14:19:27.554881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:27.555049 392 external_mini_cluster.cc:1428] Setting key fd8a58b1200fccc7c28b14cd8fbbb094
I20260620 14:19:27.558213 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.549830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34325 (local address 127.0.98.62:43329)
0620 14:19:27.549872 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.549874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.550051 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:19:27.550548 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.550549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.550562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.550627 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.551600 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.552103 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.553115 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.553298 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.553356 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.555646 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.555647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.555648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.555655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.556851 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.557261 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.557262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.557263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.557319 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.557577 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.557578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.557578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.557686 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.557762 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.558035 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.558113 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:19:27.558413 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.549815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:34325)
0620 14:19:27.549949 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.549984 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.550442 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.550682 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.550685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.551244 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.551447 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.551452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.552195 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.552196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.553031 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.553033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.553375 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.553375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.553479 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.553946 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.553955 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.555505 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.556941 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.556947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.556949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.557172 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.557395 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.557396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.557396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.557503 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.557752 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.557758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.557903 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.558018 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.558179 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260620 14:19:27.558938 6948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260620 14:19:27.559017 6948 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:27.559219 6948 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260620 14:19:27.559672 6488 ts_manager.cc:194] Registered new tserver with Master: d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3:39473)
I20260620 14:19:27.560067 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34325
I20260620 14:19:27.567008 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 14:19:27.592604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33293
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43329,127.0.98.61:33293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:27.673029 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:27.673273 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:27.673313 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:27.674459 6953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:27.674530 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:27.674554 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:27.674582 6953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:27.674614 6953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:27.676074 6953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43329,127.0.98.61:33293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
W20260620 14:19:27.676214 6953 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43329,127.0.98.61:33293'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:19:27.676373 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:27.676589 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:27.681373 6953 server_base.cc:1034] running on GCE node
W20260620 14:19:27.681344 6959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.681344 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.681335 6958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.681789 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:27.682089 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:27.683260 6953 hybrid_clock.cc:648] HybridClock initialized: now 1781965167683244 us; error 52 us; skew 500 ppm
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:27.685655 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:19:27.686152 6953 webserver.cc:460] Webserver started at http://127.0.98.61:37613/ using document root <none> and password file <none>
I20260620 14:19:27.686378 6953 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:27.686445 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:27.686594 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:27.687726 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/instance:
uuid: "04049bf957fc4fa4836fb4abe2149c97"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "43912139d85cc7938ab15408f8a8f480"
server_key_iv: "66a0af103d28dc22442e489ef9f2ca4f"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.688073 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal/instance:
uuid: "04049bf957fc4fa4836fb4abe2149c97"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "43912139d85cc7938ab15408f8a8f480"
server_key_iv: "66a0af103d28dc22442e489ef9f2ca4f"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.689384 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:27.690160 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.690333 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:27.690436 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
uuid: "04049bf957fc4fa4836fb4abe2149c97"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "43912139d85cc7938ab15408f8a8f480"
server_key_iv: "66a0af103d28dc22442e489ef9f2ca4f"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.690513 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:27.707970 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:27.709978 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:27.710135 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:27.714126 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33293
I20260620 14:19:27.714161 7020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33293 every 8 connection(s)
I20260620 14:19:27.714516 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:19:27.715453 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:27.717137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 6953
I20260620 14:19:27.716996 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:19:27.726846 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.717743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.0.98.62:43329)
0620 14:19:27.717791 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.717793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.718029 (+ 236us) server_negotiation.cc:407] Connection header received
0620 14:19:27.718477 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.718477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.718490 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.718570 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.719135 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.719700 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.720810 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.721102 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.721151 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.723517 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.723519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.723519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.723528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.725283 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.725716 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.725717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.725718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.725778 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.726083 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.726084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.726085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.726278 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.726350 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.726661 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.726739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:27.727249 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.717727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.0.98.61:33293)
0620 14:19:27.717937 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.717947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.717976 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:19:27.718372 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.718378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.718397 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.718504 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.719117 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.720204 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.721010 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.721376 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.721449 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.724087 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.724093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.724094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.724100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.725852 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.726223 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.726226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.726226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.726277 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.726484 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.726485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.726485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.726627 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.726759 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.726855 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.726926 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260620 14:19:27.727252 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.717727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57444)
0620 14:19:27.717901 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.717939 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.718388 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.718597 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.718600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.718883 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.719035 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.719039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.719795 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.719796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.720736 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.720738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.721170 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.721171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.721279 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.721842 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.721849 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.723427 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.725374 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.725380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.725381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.725638 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.725863 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.725864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.725865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.726015 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.726360 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.726366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.726532 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.726613 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.726775 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260620 14:19:27.727903 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:19:27.734176 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.728143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.0.1:33254)
0620 14:19:27.728188 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.728199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.728244 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.728370 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.728371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.728498 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.728531 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.728532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.729132 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.729133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.729638 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.729640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.730015 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.730016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.730075 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.730500 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.730504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.731690 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.733234 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.733235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.733235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.733393 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.733597 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.733598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.733598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.733648 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.733875 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.733876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.733910 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.733981 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.734072 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:19:27.734220 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.728197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.0.98.61:33293)
0620 14:19:27.728225 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.728227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.728234 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:27.728298 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.728299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.728308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.728367 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.728600 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.729048 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.729733 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.729937 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.730015 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.731767 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.731768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.731768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.731774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.733143 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.733485 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.733486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.733487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.733537 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.733717 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.733718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.733718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.733814 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.733885 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.734002 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.734118 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":14}
I20260620 14:19:27.735533 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Bootstrap starting.
I20260620 14:19:27.736335 7021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:27.736563 7021 log.cc:824] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:27.737238 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: No bootstrap required, opened a new log
I20260620 14:19:27.738586 7021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } }
I20260620 14:19:27.738711 7021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:27.738734 7021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04049bf957fc4fa4836fb4abe2149c97, State: Initialized, Role: FOLLOWER
I20260620 14:19:27.738838 7021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } }
I20260620 14:19:27.739149 7021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:27.739120 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } } }
I20260620 14:19:27.739225 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:19:27.740943 7038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:19:27.741014 7038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:19:27.742492 6953 master_runner.cc:428] Detected that this master 04049bf957fc4fa4836fb4abe2149c97 is joining an existing cluster
I20260620 14:19:27.742525 6953 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33293
I20260620 14:19:27.750001 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.743602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.0.98.62:43329)
0620 14:19:27.743647 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.743649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.743705 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:19:27.743780 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.743781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.743790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.743856 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.744296 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.744878 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.745736 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.745937 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.746019 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.747147 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.747148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.747148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.747154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.748780 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.749244 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.749245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.749245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.749298 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.749501 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.749502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.749502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.749614 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.749688 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.749839 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.749911 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:19:27.750015 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.743494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57454)
0620 14:19:27.743650 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.743661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.743715 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.743912 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.743913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.744118 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.744215 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.744216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.744940 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.744941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.745645 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.745647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.746019 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.746020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.746083 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.746552 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.746557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.746848 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.748879 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.748880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.748881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.749162 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.749377 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.749377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.749378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.749434 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.749680 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.749681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.749733 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.749823 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.749912 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260620 14:19:27.750560 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.743868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.0.98.61:33293)
0620 14:19:27.743899 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.743901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.743966 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:19:27.744009 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.744010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.744019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.744092 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.744585 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.745345 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.746381 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.746649 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.746686 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.747639 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.747640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.747640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.747646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.749321 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.749671 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.749671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.749672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.749718 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.749886 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.749887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.749887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.749989 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.750028 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.750405 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.750478 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:19:27.750833 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.743718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.0.1:33260)
0620 14:19:27.743867 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.743875 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.743958 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.744161 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.744162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.744432 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.744493 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.744495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.745433 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.745434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.746282 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.746284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.746750 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.746750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.746805 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.747155 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.747160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.747514 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.749405 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.749405 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.749406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.749595 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.749779 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.749780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.749780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.749826 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.750254 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.750255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.750301 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.750661 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.750739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:19:27.759356 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.752255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.0.1:33264)
0620 14:19:27.752365 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.752405 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.752474 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.752626 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.752628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.752857 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.752896 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.752897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.753729 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.753730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.754303 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.754305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.754745 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.754746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.754796 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.755276 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.755282 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.756630 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.758304 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.758310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.758312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.758516 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.758712 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.758713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.758713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.758777 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.759069 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.759076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.759129 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.759196 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.759279 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":69,"threads_started":1}
I20260620 14:19:27.759400 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.752307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.0.98.61:33293)
0620 14:19:27.752347 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.752349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.752472 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:19:27.752531 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.752532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.752544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.752626 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.752963 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.753653 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.754379 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.754673 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.754742 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.756704 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.756706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.756706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.756713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.758220 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.758587 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.758588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.758588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.758646 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.758835 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.758836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.758836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.758956 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.759024 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.759235 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.759310 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:19:27.759924 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33293: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33293
}
attrs {
promote: true
}
}
dest_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456"
cas_config_opid_index: -1
I20260620 14:19:27.760244 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } }
I20260620 14:19:27.760703 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 04049bf957fc4fa4836fb4abe2149c97 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } }
I20260620 14:19:27.760983 6995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:27.761667 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33293
I20260620 14:19:27.761896 6995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Refusing update from remote peer 10c59e1e1ea54a74ad3fc0f3f761c456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:19:27.761910 6953 master.cc:537] Master@127.0.98.61:33293 shutting down...
I20260620 14:19:27.761904 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } }
I20260620 14:19:27.761981 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:27.762200 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:19:27.762907 6473 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:19:27.763363 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 -> Peer 04049bf957fc4fa4836fb4abe2149c97 (127.0.98.61:33293): Couldn't send request to peer 04049bf957fc4fa4836fb4abe2149c97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:19:27.764967 6953 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:27.765053 6953 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:27.765089 6953 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: stopping tablet replica
I20260620 14:19:27.777505 6953 master.cc:552] Master@127.0.98.61:33293 shutdown complete.
I20260620 14:19:27.777550 6953 master_runner.cc:305] Clearing existing system tablet
I20260620 14:19:27.777697 6953 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:27.777736 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:27.778421 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260620 14:19:27.778961 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.779072 6953 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:27.779127 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.779150 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
uuid: "04049bf957fc4fa4836fb4abe2149c97"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "43912139d85cc7938ab15408f8a8f480"
server_key_iv: "66a0af103d28dc22442e489ef9f2ca4f"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.779831 6953 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:27.781313 6953 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:19:27.781404 6953 log.cc:1196] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:19:27.781608 6953 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Deleting consensus metadata
I20260620 14:19:27.781734 6953 master_runner.cc:315] Copying system tablet from 127.0.98.62:43329
I20260620 14:19:27.782382 6953 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43329
I20260620 14:19:27.788600 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.782727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.0.98.62:43329)
0620 14:19:27.782774 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.782777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.782842 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:19:27.782922 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.782923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.782932 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.782989 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.783313 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.783921 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.784692 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.784898 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.784985 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.785953 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.785954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.785955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.785962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.787345 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.787819 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.787820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.787821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.787884 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.788082 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.788083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.788083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.788196 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.788286 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.788437 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.788509 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:19:27.788614 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.782661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57456)
0620 14:19:27.782776 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:27.782787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:27.782859 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:27.783001 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:27.783001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:27.783174 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:27.783225 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.783227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.784005 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.784007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.784585 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:27.784587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.784985 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.784986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:27.785058 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.785551 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:27.785555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:27.785843 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:27.787448 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.787449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.787449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.787695 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.787951 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:27.787952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:27.787952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:27.788010 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:27.788279 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:27.788279 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:27.788323 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:19:27.788411 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:19:27.788509 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260620 14:19:27.789101 6509 tablet_copy_service.cc:140] P 10c59e1e1ea54a74ad3fc0f3f761c456: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04049bf957fc4fa4836fb4abe2149c97 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57456)
I20260620 14:19:27.789187 6509 tablet_copy_service.cc:161] P 10c59e1e1ea54a74ad3fc0f3f761c456: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04049bf957fc4fa4836fb4abe2149c97 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57456: session id = 04049bf957fc4fa4836fb4abe2149c97-00000000000000000000000000000000
I20260620 14:19:27.789803 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:19:27.790380 6953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:19:27.791234 6953 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: tablet copy: Starting download of 0 data blocks...
I20260620 14:19:27.791380 6953 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: tablet copy: Starting download of 1 WAL segments...
I20260620 14:19:27.792377 6953 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:19:27.793143 6509 tablet_copy_service.cc:342] P 10c59e1e1ea54a74ad3fc0f3f761c456: Request end of tablet copy session 04049bf957fc4fa4836fb4abe2149c97-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57456
I20260620 14:19:27.793200 6509 tablet_copy_service.cc:433] P 10c59e1e1ea54a74ad3fc0f3f761c456: ending tablet copy session 04049bf957fc4fa4836fb4abe2149c97-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04049bf957fc4fa4836fb4abe2149c97
I20260620 14:19:27.795647 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:27.796768 7063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:27.796886 7061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.796917 6953 server_base.cc:1034] running on GCE node
W20260620 14:19:27.796784 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:27.797128 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:27.797281 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:27.798420 6953 hybrid_clock.cc:648] HybridClock initialized: now 1781965167798412 us; error 23 us; skew 500 ppm
Jun 20 14:19:27 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:27.799790 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:19:27.800081 6953 webserver.cc:460] Webserver started at http://127.0.98.61:38411/ using document root <none> and password file <none>
I20260620 14:19:27.800215 6953 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:27.800254 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:27.800892 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:27.801422 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:27.801540 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:27.801584 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
uuid: "04049bf957fc4fa4836fb4abe2149c97"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "43912139d85cc7938ab15408f8a8f480"
server_key_iv: "66a0af103d28dc22442e489ef9f2ca4f"
server_key_version: "encryptionkey@0"
I20260620 14:19:27.801636 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:27.808497 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:27.810001 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:27.813105 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33293
I20260620 14:19:27.813416 7122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33293 every 8 connection(s)
I20260620 14:19:27.813555 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:19:27.814394 7123 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:19:27.814901 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Bootstrap starting.
I20260620 14:19:27.817706 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:19:27.817951 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Bootstrap complete.
I20260620 14:19:27.818177 7123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } }
I20260620 14:19:27.818271 7123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04049bf957fc4fa4836fb4abe2149c97, State: Initialized, Role: LEARNER
I20260620 14:19:27.818323 7123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } }
I20260620 14:19:27.818488 7123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:27.818512 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } }
I20260620 14:19:27.818645 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:19:27.820251 7137 catalog_manager.cc:1259] Loaded cluster ID: eb1a048fae624342b5aa6e9d54fc02fa
I20260620 14:19:27.820307 7137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: loading cluster ID for follower catalog manager: success
I20260620 14:19:27.821556 7137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: acquiring CA information for follower catalog manager: success
I20260620 14:19:27.821926 7137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:19:27.836115 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:27.828922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.98.61:33293)
0620 14:19:27.829043 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:27.829045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:27.829054 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:27.829079 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:27.829080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:27.829092 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:27.829132 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:27.829843 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.830566 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.831342 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:27.831587 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:27.831666 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:27.833002 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:27.833004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:27.833004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:27.833012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:27.834690 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.835178 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.835179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.835180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.835244 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:27.835460 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:27.835461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:27.835461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:27.835654 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:27.835764 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:27.835930 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:19:27.836008 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.60.keytab.
W20260620 14:19:27.951654 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.60: real 0.115s user 0.000s sys 0.003s
I20260620 14:19:27.952000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38525
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:28.035200 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:28.035403 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:28.035425 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:28.036651 7141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:28.036695 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:28.036708 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:28.036721 7141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:28.036736 7141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:28.038121 7141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:28.038302 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:28.038463 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:28.043318 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.043323 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:28.043340 7141 server_base.cc:1034] running on GCE node
W20260620 14:19:28.043300 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:28.043889 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:28.044198 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:28.045367 7141 hybrid_clock.cc:648] HybridClock initialized: now 1781965168045355 us; error 39 us; skew 500 ppm
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:28.047886 7141 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:19:28.048316 7141 webserver.cc:460] Webserver started at http://127.0.98.60:42701/ using document root <none> and password file <none>
I20260620 14:19:28.048544 7141 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:28.048614 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:28.048768 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:28.049832 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/instance:
uuid: "68ac82958a264fdda67bdc8eb238c2f0"
format_stamp: "Formatted at 2026-06-20 14:19:28 on dist-test-slave-h6n5"
server_key: "3c837c912a6d61a484a8bb30a9133863"
server_key_iv: "d1d69716304d67ce9f96bc00701e4848"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.050203 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal/instance:
uuid: "68ac82958a264fdda67bdc8eb238c2f0"
format_stamp: "Formatted at 2026-06-20 14:19:28 on dist-test-slave-h6n5"
server_key: "3c837c912a6d61a484a8bb30a9133863"
server_key_iv: "d1d69716304d67ce9f96bc00701e4848"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.051601 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:28.052335 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.052520 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:28.052618 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
uuid: "68ac82958a264fdda67bdc8eb238c2f0"
format_stamp: "Formatted at 2026-06-20 14:19:28 on dist-test-slave-h6n5"
server_key: "3c837c912a6d61a484a8bb30a9133863"
server_key_iv: "d1d69716304d67ce9f96bc00701e4848"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.052706 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:28.061965 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:28.063897 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:28.064056 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:28.068282 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38525
I20260620 14:19:28.068316 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38525 every 8 connection(s)
I20260620 14:19:28.068673 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:19:28.069707 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:28.071805 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:19:28.076658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 7141
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:28.081343 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.072590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.0.98.62:43329)
0620 14:19:28.072663 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.072665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.072815 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:19:28.073272 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.073273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.073283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.073341 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.073978 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.074462 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.075628 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.075884 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.075949 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.078350 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.078352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.078352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.078360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.079947 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.080313 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.080314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.080315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.080379 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.080697 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.080699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.080699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.080821 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.080901 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.081171 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.081252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:19:28.081543 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.072507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57460)
0620 14:19:28.072682 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.072718 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.073162 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.073430 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.073433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.073694 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.073892 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.073896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.074542 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.074543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.075465 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.075468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.075965 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.075966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.076098 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.076638 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.076646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.078265 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.080022 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.080027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.080029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.080242 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.080460 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.080461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.080462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.080621 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.080892 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.080897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.081038 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.081150 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.081299 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260620 14:19:28.082152 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:19:28.085901 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.077160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.0.98.60:38525)
0620 14:19:28.077308 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.077312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.077335 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:19:28.077353 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.077355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.077370 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.077460 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.078040 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.079007 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.079864 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.080259 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.080347 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.082947 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.082953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.082954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.082960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.084580 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.084911 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.084915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.084915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.084982 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.085211 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.085211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.085212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.085415 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.085572 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.085664 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.085739 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:19:28.089349 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.082426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.0.1:33298)
0620 14:19:28.082499 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.082511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.082559 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.082675 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.082676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.082817 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.082871 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.082873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.083975 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.083975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.084479 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.084481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.084958 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.084959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.085038 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.085522 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.085527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.086851 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.088410 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.088410 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.088411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.088579 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.088792 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.088793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.088793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.088843 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.089124 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.089125 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.089166 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.089218 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.089288 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:19:28.089386 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.082465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.0.98.61:33293)
0620 14:19:28.082504 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.082506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.082546 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:19:28.082617 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.082618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.082627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.082670 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.082946 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.083700 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.084588 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.084857 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.084949 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.086971 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.086972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.086972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.086979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.088286 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.088653 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.088661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.088661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.088721 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.088939 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.088940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.088941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.089060 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.089122 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.089236 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.089325 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:19:28.089910 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:19:28.096058 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.090189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.0.1:38828)
0620 14:19:28.090230 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.090247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.090279 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.090386 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.090386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.090487 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.090515 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.090516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.091145 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.091146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.091684 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.091686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.092062 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.092063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.092100 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.092446 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.092450 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.093542 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.095145 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.095146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.095147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.095306 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.095502 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.095503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.095503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.095600 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.095804 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.095805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.095864 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.095917 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.095993 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:19:28.096174 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.090238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.0.98.60:38525)
0620 14:19:28.090270 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.090272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.090279 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:28.090333 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.090333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.090342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.090384 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.090592 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.091053 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.091774 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.091993 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.092051 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.093615 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.093616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.093616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.093622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.095062 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.095384 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.095385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.095385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.095440 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.095652 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.095653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.095653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.095744 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.095794 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.095965 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.096040 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":15}
I20260620 14:19:28.097412 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Bootstrap starting.
I20260620 14:19:28.098248 7209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:28.098526 7209 log.cc:824] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:28.099246 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: No bootstrap required, opened a new log
I20260620 14:19:28.100701 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } }
I20260620 14:19:28.100840 7209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:28.100886 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ac82958a264fdda67bdc8eb238c2f0, State: Initialized, Role: FOLLOWER
I20260620 14:19:28.101037 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } }
I20260620 14:19:28.101330 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:28.101293 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } } }
I20260620 14:19:28.101410 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:19:28.103087 7225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:19:28.103160 7225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:19:28.105150 7141 master_runner.cc:428] Detected that this master 68ac82958a264fdda67bdc8eb238c2f0 is joining an existing cluster
I20260620 14:19:28.105201 7141 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:38525
I20260620 14:19:28.112834 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.106676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.98.61:33293)
0620 14:19:28.106715 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.106716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.106725 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:28.106746 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.106747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.106756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.106836 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.107603 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.108087 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.108813 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.109014 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.109098 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.110067 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.110068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.110069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.110075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.111625 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.112053 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.112054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.112054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.112109 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.112308 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.112308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.112309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.112425 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.112496 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.112660 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.112745 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:19:28.112843 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.106356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.0.1:33300)
0620 14:19:28.106640 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.106649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.106692 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.106885 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.106886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.107036 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.107493 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.107495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.108168 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.108169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.108726 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.108728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.109109 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.109110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.109175 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.109647 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.109652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.109961 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.111726 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.111727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.111728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.111973 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.112180 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.112181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.112181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.112238 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.112501 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.112502 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.112556 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.112624 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.112717 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":62,"threads_started":1}
I20260620 14:19:28.117561 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.106528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.0.1:38840)
0620 14:19:28.111649 (+ 5121us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.111660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.111735 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.111868 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.111869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.112092 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.112451 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.112454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.113235 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.113236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.113895 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.113897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.114264 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.114265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.114322 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.114748 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.114752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.115071 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.116531 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.116531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.116532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.116687 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.116884 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.116884 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.116885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.116943 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.117209 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.117210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.117248 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.117315 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.117425 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5100,"thread_start_us":67,"threads_started":1}
I20260620 14:19:28.117570 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.106555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.0.98.60:38525)
0620 14:19:28.106593 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.106594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.111712 (+ 5118us) server_negotiation.cc:407] Connection header received
0620 14:19:28.111786 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.111787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.111797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.111850 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.112547 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.113157 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.113992 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.114184 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.114264 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.115164 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.115165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.115166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.115171 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.116439 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.116769 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.116770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.116770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.116821 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.117020 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.117021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.117021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.117144 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.117218 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.117342 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.117487 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":18}
I20260620 14:19:28.131233 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.106203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.0.98.62:43329)
0620 14:19:28.106259 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.106262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.106337 (+ 75us) server_negotiation.cc:407] Connection header received
0620 14:19:28.106415 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.106416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.106427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.106489 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.106898 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.107710 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.108845 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.109138 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.109179 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.128680 (+ 19501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.128682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.128683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.128690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.130133 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.130529 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.130530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.130530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.130579 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.130772 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.130772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.130772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.130876 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.130958 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.131106 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.131166 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:19:28.131310 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.106130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57462)
0620 14:19:28.106276 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.106287 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.106346 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.106492 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.106493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.106716 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.106796 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.106798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.107807 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.107808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.108740 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.108742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.109255 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.109256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.109318 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.128090 (+ 18772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.128096 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.128550 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.130222 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.130223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.130223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.130452 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.130642 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.130643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.130644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.130709 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.130957 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.130958 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.131010 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.131086 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.131193 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:19:28.137130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6551
I20260620 14:19:28.140786 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.133443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.0.1:38842)
0620 14:19:28.133487 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.133497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.133532 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.133696 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.133697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.133841 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.133889 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.133891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.134525 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.134526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.135036 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.135037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.135459 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.135460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.135525 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.135948 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.135953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.137337 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.139297 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.139298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.139298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.139668 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.139964 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.139964 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.139965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.140050 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.140410 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.140411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.140479 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.140580 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.140684 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:19:28.141095 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.133519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.0.98.60:38525)
0620 14:19:28.133573 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.133575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.133583 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:28.133612 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.133613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.133622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.133694 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.133963 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.134435 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.135115 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.135374 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.135459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.137431 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.137433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.137433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.137439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.139213 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.139778 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.139779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.139780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.139860 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.140148 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.140149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.140150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.140349 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.140415 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.140646 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.140742 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:19:28.144161 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:38525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 38525
}
attrs {
promote: true
}
}
dest_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456"
cas_config_opid_index: 5
I20260620 14:19:28.144258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35681
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=35411
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:19:28.144440 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } }
I20260620 14:19:28.144814 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.145797 7047 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:38525
I20260620 14:19:28.145875 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.145964 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:28.146170 7183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:28.146680 7141 master.cc:537] Master@127.0.98.60:38525 shutting down...
I20260620 14:19:28.147213 7183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Refusing update from remote peer 10c59e1e1ea54a74ad3fc0f3f761c456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:19:28.148046 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:19:28.149017 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 -> Peer 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60:38525): Couldn't send request to peer 68ac82958a264fdda67bdc8eb238c2f0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:19:28.150087 7141 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:28.150187 7141 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:28.150249 7141 tablet_replica.cc:330] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: stopping tablet replica
I20260620 14:19:28.151683 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.145658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.0.1:33304)
0620 14:19:28.146284 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.146295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.146346 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.146601 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.146603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.146604 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:28.146642 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.146644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.147658 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.147659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.150048 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.150051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.151279 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.151280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.151477 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.151499 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.151572 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.151614 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612}
I20260620 14:19:28.152117 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.145750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.0.98.61:33293)
0620 14:19:28.146054 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.146056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.146399 (+ 343us) server_negotiation.cc:407] Connection header received
0620 14:19:28.146426 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.146427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.146449 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.146545 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:28.146739 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.147542 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.150214 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.151100 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.151169 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.151245 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.152015 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.152054 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260620 14:19:28.152294 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 LEARNER]: Refusing update from remote peer 10c59e1e1ea54a74ad3fc0f3f761c456: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:19:28.152468 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:19:28.152989 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.154079 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.154251 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:19:28.154278 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c59e1e1ea54a74ad3fc0f3f761c456. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: true } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.154378 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:19:28.154938 7047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: attempting to promote NON_VOTER 04049bf957fc4fa4836fb4abe2149c97 to VOTER
I20260620 14:19:28.155161 7047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } }
I20260620 14:19:28.155783 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 LEARNER]: Refusing update from remote peer 10c59e1e1ea54a74ad3fc0f3f761c456: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:19:28.155905 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.155627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.0.1:38844)
0620 14:19:28.155837 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.155877 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 14:19:28.156054 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:19:28.156296 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.156110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.0.1:38854)
0620 14:19:28.156251 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.156273 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260620 14:19:28.156418 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 -> Peer 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60:38525): Couldn't send request to peer 68ac82958a264fdda67bdc8eb238c2f0. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:19:28.156737 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04049bf957fc4fa4836fb4abe2149c97 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.156890 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04049bf957fc4fa4836fb4abe2149c97 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.157709 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.157778 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:19:28.157788 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.157855 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:28.157895 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.157969 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:28.163846 7141 master.cc:552] Master@127.0.98.60:38525 shutdown complete.
I20260620 14:19:28.163897 7141 master_runner.cc:305] Clearing existing system tablet
I20260620 14:19:28.164026 7141 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:28.164072 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:28.164815 7141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:28.165369 7238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.165483 7141 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:28.165529 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:28.165549 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
uuid: "68ac82958a264fdda67bdc8eb238c2f0"
format_stamp: "Formatted at 2026-06-20 14:19:28 on dist-test-slave-h6n5"
server_key: "3c837c912a6d61a484a8bb30a9133863"
server_key_iv: "d1d69716304d67ce9f96bc00701e4848"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.166324 7141 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:28.167734 7141 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:19:28.167796 7141 log.cc:1196] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:19:28.167986 7141 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Deleting consensus metadata
I20260620 14:19:28.168180 7141 master_runner.cc:315] Copying system tablet from 127.0.98.62:43329
I20260620 14:19:28.168766 7141 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43329
I20260620 14:19:28.176474 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.169107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.0.98.62:43329)
0620 14:19:28.169158 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.169160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.169220 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:19:28.169291 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.169292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.169321 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.169374 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.169787 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.170608 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.171700 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.172001 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.172072 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.173549 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.173550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.173551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.173559 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.175220 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.175649 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.175650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.175651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.175712 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.175927 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.175928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.175928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.176062 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.176133 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.176302 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.176385 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:19:28.176481 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.169037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57478)
0620 14:19:28.169150 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.169160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.169217 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.169433 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.169434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.169637 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.169697 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.169700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.170700 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.170701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.171617 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.171620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.172080 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.172080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.172134 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.173071 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.173076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.173460 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.175308 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.175309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.175309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.175530 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.175783 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.175784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.175784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.175855 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.176134 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.176135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.176189 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.176262 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.176365 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260620 14:19:28.177053 6509 tablet_copy_service.cc:140] P 10c59e1e1ea54a74ad3fc0f3f761c456: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 68ac82958a264fdda67bdc8eb238c2f0 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57478)
I20260620 14:19:28.177109 6509 tablet_copy_service.cc:161] P 10c59e1e1ea54a74ad3fc0f3f761c456: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 68ac82958a264fdda67bdc8eb238c2f0 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57478: session id = 68ac82958a264fdda67bdc8eb238c2f0-00000000000000000000000000000000
I20260620 14:19:28.177675 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:19:28.178104 7141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:19:28.179270 7141 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: tablet copy: Starting download of 0 data blocks...
I20260620 14:19:28.179419 7141 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: tablet copy: Starting download of 1 WAL segments...
I20260620 14:19:28.180186 7141 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:19:28.181100 6509 tablet_copy_service.cc:342] P 10c59e1e1ea54a74ad3fc0f3f761c456: Request end of tablet copy session 68ac82958a264fdda67bdc8eb238c2f0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57478
I20260620 14:19:28.181151 6509 tablet_copy_service.cc:433] P 10c59e1e1ea54a74ad3fc0f3f761c456: ending tablet copy session 68ac82958a264fdda67bdc8eb238c2f0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 68ac82958a264fdda67bdc8eb238c2f0
I20260620 14:19:28.183781 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:28.184978 7245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.188459 7246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.188676 7248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:28.191726 7141 server_base.cc:1034] running on GCE node
I20260620 14:19:28.191850 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:28.192009 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:28.193152 7141 hybrid_clock.cc:648] HybridClock initialized: now 1781965168193126 us; error 45 us; skew 500 ppm
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:28.194671 7141 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:19:28.194938 7141 webserver.cc:460] Webserver started at http://127.0.98.60:39521/ using document root <none> and password file <none>
I20260620 14:19:28.195063 7141 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:28.195103 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:28.195892 7141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:28.196491 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.196599 7141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:28.196633 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
uuid: "68ac82958a264fdda67bdc8eb238c2f0"
format_stamp: "Formatted at 2026-06-20 14:19:28 on dist-test-slave-h6n5"
server_key: "3c837c912a6d61a484a8bb30a9133863"
server_key_iv: "d1d69716304d67ce9f96bc00701e4848"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.196689 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:28.220664 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:28.222579 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:28.225713 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38525
I20260620 14:19:28.226046 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:19:28.227766 7307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38525 every 8 connection(s)
I20260620 14:19:28.228722 7308 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:19:28.229336 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Bootstrap starting.
I20260620 14:19:28.232628 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:19:28.232867 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: Bootstrap complete.
I20260620 14:19:28.233064 7308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } }
I20260620 14:19:28.233166 7308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 68ac82958a264fdda67bdc8eb238c2f0, State: Initialized, Role: LEARNER
I20260620 14:19:28.233208 7308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } }
I20260620 14:19:28.233387 7308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:28.233409 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.233616 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:19:28.235001 7322 catalog_manager.cc:1259] Loaded cluster ID: eb1a048fae624342b5aa6e9d54fc02fa
I20260620 14:19:28.235705 7322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: loading cluster ID for follower catalog manager: success
I20260620 14:19:28.236788 7322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: acquiring CA information for follower catalog manager: success
I20260620 14:19:28.237074 7322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:19:28.248871 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:28.249145 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:28.249185 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:28.250365 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:28.250435 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:28.250483 7234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:28.252027 7234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35411
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:28.252281 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:28.252514 7234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:28.258234 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.258299 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.258252 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:28.258445 7234 server_base.cc:1034] running on GCE node
I20260620 14:19:28.258701 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:28.258962 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:28.260123 7234 hybrid_clock.cc:648] HybridClock initialized: now 1781965168260094 us; error 45 us; skew 500 ppm
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:28.262574 7234 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:19:28.263011 7234 webserver.cc:460] Webserver started at http://127.0.98.1:35411/ using document root <none> and password file <none>
I20260620 14:19:28.263228 7234 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:28.263307 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:28.264694 7234 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:28.265254 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.265419 7234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:28.265518 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "acf0c092f8ac435da3112368a2180822"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "49fdefc70af6ec5d6f95510b144872f1"
server_key_iv: "469f5e756ed71b197710248ebba528ad"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.265865 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:28.291010 7234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:28.292896 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:28.293061 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:28.293299 7234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:28.293605 7234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:28.293658 7234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.293696 7234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:28.293723 7234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.299721 7234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35681
I20260620 14:19:28.299832 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35681 every 8 connection(s)
I20260620 14:19:28.300151 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:28.301096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 7234
I20260620 14:19:28.301256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6685
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:19:28.310907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39859
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=38255
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:19:28.313808 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.301263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33357 (local address 127.0.98.62:43329)
0620 14:19:28.301320 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.301322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.301455 (+ 133us) server_negotiation.cc:407] Connection header received
0620 14:19:28.302112 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.302113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.302126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.302158 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.303192 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.303993 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.305954 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.306257 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.306302 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.310083 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.310085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.310085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.310093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.312083 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.312728 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.312729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.312729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.312789 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.313068 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.313069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.313069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.313205 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.313250 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.313618 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.313697 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:19:28.314029 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.301314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.98.1:43865)
0620 14:19:28.301511 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.301518 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.301997 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.302464 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.302465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.302835 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.303082 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.303088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.304402 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.304403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.305850 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.305853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.306392 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.306392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.306543 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.307317 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.307323 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.309943 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.312161 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.312168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.312170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.312629 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.312848 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.312848 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.312849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.312986 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.313293 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.313299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.313463 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.313573 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.313756 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":175,"thread_start_us":67,"threads_started":1}
I20260620 14:19:28.314102 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.300861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55695 (local address 127.0.98.61:33293)
0620 14:19:28.300946 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.300948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.301341 (+ 393us) server_negotiation.cc:407] Connection header received
0620 14:19:28.302112 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.302113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.302126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.302158 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.303191 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.303983 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.305952 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.306268 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.306304 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.310153 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.310154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.310154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.310161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.312934 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.313289 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.313290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.313290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.313346 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.313535 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.313536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.313536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.313685 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.313724 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.313942 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.314013 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:19:28.314429 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.300860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.98.1:55695)
0620 14:19:28.301063 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.301100 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.301988 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.302302 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.302306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.302771 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.303084 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.303088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.304115 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.304116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.305846 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.305849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.306392 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.306393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.306543 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.307237 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.307247 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.310064 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.313016 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.313016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.313017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.313207 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.313413 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.313413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.313414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.313465 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.313783 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.313784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.313822 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.314228 (+ 406us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.314328 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":133,"thread_start_us":86,"threads_started":1}
I20260620 14:19:28.314522 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.302167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43865 (local address 127.0.98.60:38525)
0620 14:19:28.302296 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.302298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.302306 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:28.302332 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.302333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.302345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.302406 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.303185 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.304309 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.305942 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.306269 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.306333 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.310049 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.310051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.310051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.310058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.312061 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.312716 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.312717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.312717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.312780 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.313056 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.313057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.313058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.313198 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.313267 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.314272 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.314366 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260620 14:19:28.314855 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33293
I20260620 14:19:28.314953 7453 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.315153 7453 heartbeater.cc:511] Master 127.0.98.61:33293 requested a full tablet report, sending...
I20260620 14:19:28.315347 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38525
I20260620 14:19:28.315402 7451 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.315476 7451 heartbeater.cc:511] Master 127.0.98.60:38525 requested a full tablet report, sending...
I20260620 14:19:28.315472 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.301157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:33357)
0620 14:19:28.301394 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.301403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.301989 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.302701 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.302702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.302955 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.303082 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.303088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.304115 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.304116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.305846 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.305849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.306390 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.306392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.306543 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.307437 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.307442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.309980 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.312183 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.312183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.312184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.312629 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.312871 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.312871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.312872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.312989 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.313318 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.313319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.313463 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.314905 (+ 1442us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.315008 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":85,"thread_start_us":67,"threads_started":1}
I20260620 14:19:28.315918 7272 ts_manager.cc:194] Registered new tserver with Master: acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681)
I20260620 14:19:28.316046 7087 ts_manager.cc:194] Registered new tserver with Master: acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681)
I20260620 14:19:28.316339 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260620 14:19:28.316395 7452 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.316468 7452 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260620 14:19:28.316810 6488 ts_manager.cc:194] Re-registered known tserver with Master: acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681)
I20260620 14:19:28.317193 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33357
W20260620 14:19:28.395362 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:28.395610 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:28.395661 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:28.397051 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:28.397111 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:28.397156 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:28.398689 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38255
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:28.398983 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:28.399214 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:28.404188 7459 server_base.cc:1034] running on GCE node
W20260620 14:19:28.404148 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.404176 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.404156 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:28.404621 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:28.404896 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:28.406067 7459 hybrid_clock.cc:648] HybridClock initialized: now 1781965168406055 us; error 25 us; skew 500 ppm
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:28.408682 7459 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:19:28.409050 7459 webserver.cc:460] Webserver started at http://127.0.98.2:38255/ using document root <none> and password file <none>
I20260620 14:19:28.409219 7459 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:28.409271 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:28.410537 7459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:28.411089 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.411271 7459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:28.411358 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "a063836e372f4021b8f5f7d65ff18c6b"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "ae65a03ea4c60f04f6fb41eeb601c393"
server_key_iv: "5d788827bad3c2646901289f5c6af755"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.411744 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:28.420814 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:28.422710 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:28.422818 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:28.422991 7459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:28.423233 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:28.423261 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.423282 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:28.423295 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.428982 7459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39859
I20260620 14:19:28.429056 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39859 every 8 connection(s)
I20260620 14:19:28.429401 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:19:28.436590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 7459
I20260620 14:19:28.436738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6819
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:28.441223 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.430483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33467 (local address 127.0.98.61:33293)
0620 14:19:28.430538 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.430540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.430553 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:19:28.431311 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.431312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.431323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.431390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.432258 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.433005 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.434222 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.434467 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.434528 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.438089 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.438091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.438091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.438098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.439773 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.440178 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.440179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.440180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.440233 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.440527 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.440528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.440528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.440673 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.440712 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.441052 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.441132 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:19:28.441299 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.430630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36993 (local address 127.0.98.60:38525)
0620 14:19:28.430673 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.430675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.430749 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:19:28.431224 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.431226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.431242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.431293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.432227 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.432744 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.434152 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.434506 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.434578 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.438070 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.438072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.438072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.438080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.440054 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.440425 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.440426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.440427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.440481 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.440674 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.440675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.440676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.440830 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.440902 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.441052 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.441150 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:19:28.441416 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.430558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.98.2:36993)
0620 14:19:28.430684 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.430694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.431073 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.431717 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.431718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.431875 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.432135 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.432140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.432863 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.432864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.434081 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.434084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.434589 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.434590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.434664 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.435303 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.435313 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.437975 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.440134 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.440135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.440136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.440345 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.440556 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.440557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.440557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.440603 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.440907 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.440908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.440948 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.441025 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.441184 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":20,"thread_start_us":60,"threads_started":1}
I20260620 14:19:28.441529 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.430409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.98.2:33467)
0620 14:19:28.430506 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.430518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.431044 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.431420 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.431420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.431850 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.432135 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.432141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.433113 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.433114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.434129 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.434131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.434531 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.434532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.434660 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.435382 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.435386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.437974 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.439838 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.439845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.439847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.440069 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.440307 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.440307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.440308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.440448 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.440760 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.440764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.440900 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.441295 (+ 395us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.441440 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":21,"thread_start_us":60,"threads_started":1}
I20260620 14:19:28.441887 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38525
I20260620 14:19:28.441964 7588 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.442099 7588 heartbeater.cc:511] Master 127.0.98.60:38525 requested a full tablet report, sending...
I20260620 14:19:28.442535 7272 ts_manager.cc:194] Registered new tserver with Master: a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2:39859)
I20260620 14:19:28.443310 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33293
I20260620 14:19:28.443353 7590 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.443398 7590 heartbeater.cc:511] Master 127.0.98.61:33293 requested a full tablet report, sending...
I20260620 14:19:28.443688 7087 ts_manager.cc:194] Registered new tserver with Master: a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2:39859)
I20260620 14:19:28.443882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39473
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=34685
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:19:28.452692 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.430138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60645 (local address 127.0.98.62:43329)
0620 14:19:28.430230 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.430232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.430405 (+ 173us) server_negotiation.cc:407] Connection header received
0620 14:19:28.431179 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.431181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.431194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.431251 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.432245 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.432995 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.434638 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.434984 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.435087 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.445517 (+ 10430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.445519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.445519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.445527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.451293 (+ 5766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.451858 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.451859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.451860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.451921 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.452185 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.452186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.452186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.452305 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.452359 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.452557 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.452624 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:19:28.453233 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.430112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:60645)
0620 14:19:28.430283 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.430318 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.431044 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.431324 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.431328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.431849 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.432135 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.432141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.433097 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.433098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.434083 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.434086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.435076 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.435077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.435139 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.435571 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.435576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.445395 (+ 9819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.451417 (+ 6022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.451418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.451419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.451724 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.452003 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.452005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.452005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.452087 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.452374 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.452375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.452441 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.452515 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.452643 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":29,"thread_start_us":97,"threads_started":1}
I20260620 14:19:28.453732 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260620 14:19:28.453806 7589 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.453888 7589 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260620 14:19:28.454195 6488 ts_manager.cc:194] Re-registered known tserver with Master: a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2:39859)
I20260620 14:19:28.454586 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60645
W20260620 14:19:28.525671 7595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:28.525866 7595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:28.525895 7595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:28.527022 7595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:28.527060 7595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:28.527087 7595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:28.528537 7595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34685
--tserver_master_addrs=127.0.98.62:43329,127.0.98.61:33293,127.0.98.60:38525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:28.528779 7595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:28.529016 7595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:28.534070 7603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:28.534129 7595 server_base.cc:1034] running on GCE node
W20260620 14:19:28.534113 7600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:28.534097 7601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:28.534541 7595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:28.534751 7595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:28.535888 7595 hybrid_clock.cc:648] HybridClock initialized: now 1781965168535859 us; error 42 us; skew 500 ppm
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:28.538318 7595 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:19:28.538767 7595 webserver.cc:460] Webserver started at http://127.0.98.3:34685/ using document root <none> and password file <none>
I20260620 14:19:28.539017 7595 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:28.539103 7595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:28.540477 7595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:28.541030 7610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.541191 7595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:28.541288 7595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "d8b0c869b06c4f3fac1d420969f912a6"
format_stamp: "Formatted at 2026-06-20 14:19:27 on dist-test-slave-h6n5"
server_key: "d7a0729b0a25e6ede8a13ee7a5919abe"
server_key_iv: "d91be37597b4a2cc8a66f0fc871a84ba"
server_key_version: "encryptionkey@0"
I20260620 14:19:28.541642 7595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:28.556481 7595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:28.558502 7595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:28.558665 7595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:28.558917 7595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:28.559226 7595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:28.559278 7595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.559324 7595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:28.559353 7595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:28.565254 7595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39473
I20260620 14:19:28.565300 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39473 every 8 connection(s)
I20260620 14:19:28.565631 7595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:28.568645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 7595
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:19:28 dist-test-slave-h6n5 krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:19:28.579855 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.566697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57231 (local address 127.0.98.60:38525)
0620 14:19:28.566742 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.566744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.566808 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:19:28.567213 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.567215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.567226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.567280 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.568306 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.569238 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.571201 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.572462 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.572533 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.576168 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.576169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.576169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.576175 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.578300 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.578745 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.578746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.578746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.578826 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.579075 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.579076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.579077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.579201 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.579256 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.579658 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.579745 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:28.580014 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.566328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.98.3:42213)
0620 14:19:28.566483 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.566523 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.567113 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.567361 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.567376 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.567976 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.568210 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.568216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.569221 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.569221 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.570627 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.570629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.571431 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.571432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.571496 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.572344 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.572349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.575139 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.578265 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.578266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.578266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.578587 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.578829 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.578830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.578831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.578978 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.579300 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.579302 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.579439 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.579546 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.579765 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":90,"thread_start_us":92,"threads_started":1}
I20260620 14:19:28.580186 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.566613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.98.3:57231)
0620 14:19:28.566744 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.566755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.567127 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.567372 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.567376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.567952 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.568209 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.568214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.569359 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.569360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.571117 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.571118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.572589 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.572590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.572661 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.574411 (+ 1750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.574416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.576079 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.578382 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.578383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.578384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.578671 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.578896 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.578896 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.578897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.578996 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.579278 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.579283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.579438 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.579545 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.580067 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1503,"thread_start_us":62,"threads_started":1}
I20260620 14:19:28.580324 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.566350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42213 (local address 127.0.98.61:33293)
0620 14:19:28.566401 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.566403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.566635 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:19:28.567218 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.567219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.567232 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.567286 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.568318 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.569100 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.571039 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.571326 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.571429 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.576402 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.576403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.576404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.576429 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.578177 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.578670 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.578671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.578671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.578729 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.579062 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.579063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.579063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.579208 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.579274 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.580141 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.580225 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:19:28.580690 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33293
I20260620 14:19:28.580808 7726 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.580792 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38525
I20260620 14:19:28.580896 7724 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.581049 7726 heartbeater.cc:511] Master 127.0.98.61:33293 requested a full tablet report, sending...
I20260620 14:19:28.581048 7724 heartbeater.cc:511] Master 127.0.98.60:38525 requested a full tablet report, sending...
I20260620 14:19:28.581511 7087 ts_manager.cc:194] Registered new tserver with Master: d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3:39473)
I20260620 14:19:28.581746 7272 ts_manager.cc:194] Registered new tserver with Master: d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3:39473)
I20260620 14:19:28.582000 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.570074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.98.61:33293)
0620 14:19:28.572021 (+ 1947us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.572023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.572030 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:28.572050 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.572050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.572060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.572113 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.573036 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.574451 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.575488 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.575840 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.575911 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.577739 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.577741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.577741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.577748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.579691 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.580135 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.580136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.580136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.580193 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.580379 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.580380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.580627 (+ 247us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.581348 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.581448 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.581789 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.581886 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1925,"thread_start_us":47,"threads_started":1}
I20260620 14:19:28.582828 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.566922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46747 (local address 127.0.98.62:43329)
0620 14:19:28.566984 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.566985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.567363 (+ 378us) server_negotiation.cc:407] Connection header received
0620 14:19:28.567762 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.567763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.567774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.567835 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.568317 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.569069 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.570709 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.570907 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.570983 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.574866 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.574868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.574868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.574876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.577910 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.579652 (+ 1742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.579653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.579653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.579713 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.581918 (+ 2205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.581919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.581921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.582063 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.582119 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.582645 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.582732 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:28.583045 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.570084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.0.98.60:38525)
0620 14:19:28.570329 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.570331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.571469 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 14:19:28.571493 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.571494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.571506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.572041 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.572861 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.573595 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.574886 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.575212 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.575280 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.576838 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.576839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.576839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.576845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.581717 (+ 4872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.582076 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.582078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.582078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.582134 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.582322 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.582323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.582323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.582424 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.582538 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.582855 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.582959 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":49,"threads_started":1}
I20260620 14:19:28.583307 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.566761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:46747)
0620 14:19:28.567309 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.567319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.567678 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.567860 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.567861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.568058 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:28.568211 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.568216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.569221 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.569221 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.570627 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.570630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.570986 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.570987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.571115 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.572092 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:28.572130 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:28.574777 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:28.578003 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.578011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.578013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.579535 (+ 1522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.579796 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:28.579796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:28.579797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:28.579849 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:28.582464 (+ 2615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:28.582464 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:28.582532 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.582595 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.582706 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":130,"thread_start_us":52,"threads_started":1}
I20260620 14:19:28.583663 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260620 14:19:28.583722 7725 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:28.583806 7725 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260620 14:19:28.584096 6488 ts_manager.cc:194] Re-registered known tserver with Master: d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3:39473)
I20260620 14:19:28.584586 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46747
I20260620 14:19:28.587672 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.569727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.0.98.62:43329)
0620 14:19:28.571395 (+ 1668us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.571397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.571403 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:19:28.571421 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.571421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.571432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.571955 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:28.574398 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.578506 (+ 4108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.583482 (+ 4976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.583816 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.583901 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.585239 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:28.585240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:28.585240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:28.585247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:28.586537 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.586857 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.586858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.586859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.586949 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:28.587148 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:28.587149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:28.587150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:28.587260 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:28.587352 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.587470 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.587572 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643,"thread_start_us":87,"threads_started":1}
I20260620 14:19:28.589035 6488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:28.590516 6488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:28.596443 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.592734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.0.1:38866)
0620 14:19:28.593005 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.593016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.593068 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.593237 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.593237 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.593239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:28.593279 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.593280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.594166 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.594167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.595234 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.595237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.596184 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.596185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.596244 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.596247 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.596292 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.596360 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":28}
I20260620 14:19:28.596508 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.592849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.0.98.60:38525)
0620 14:19:28.592895 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.592898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.593055 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:19:28.593121 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.593123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.593144 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.593233 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:28.593338 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.594055 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.595330 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.596005 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.596065 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.596133 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.596338 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.596378 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:19:28.597038 7275 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.8] Dedup: 1.7->[1.8-1.8]
I20260620 14:19:28.597818 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c59e1e1ea54a74ad3fc0f3f761c456. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.597947 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:19:28.598973 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.594396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.0.98.2:39859)
0620 14:19:28.595039 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.595044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.595053 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:28.595088 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.595091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.595109 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.595193 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:28.595305 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.596514 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.597851 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.598573 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.598655 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.598711 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.598835 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.598878 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":82,"threads_started":1}
I20260620 14:19:28.599220 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.594187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39859 (local address 127.0.0.1:54422)
0620 14:19:28.594599 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.594608 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.594645 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.595189 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.595190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.595192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:28.595223 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.595224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.596618 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.596619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.597765 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.597767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.598654 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.598655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.598721 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.598725 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.599093 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.599143 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":69,"threads_started":1}
I20260620 14:19:28.599326 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: true } } }
I20260620 14:19:28.599429 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:28.599757 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.594299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.0.98.3:39473)
0620 14:19:28.594764 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.594768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.594775 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:28.594805 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.594807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.594823 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.594916 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:28.595047 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.596102 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.598193 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.599158 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.599474 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.599592 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.599622 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.599671 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":74,"threads_started":1}
I20260620 14:19:28.599861 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.594108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39473 (local address 127.0.0.1:44606)
0620 14:19:28.594463 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.594473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.594520 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.594919 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.594920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.594922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:28.594961 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.594962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.596210 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.596211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.598097 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.598099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.599266 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.599267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.599348 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.599352 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.599380 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.599419 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":76,"threads_started":1}
I20260620 14:19:28.599963 7522 tablet_service.cc:1467] Processing CreateTablet for tablet b8c34198e4244194a1db3d8b96639fa3 (DEFAULT_TABLE table=test-table [id=590eab83c2cf4c7aa82e5c87ce681398]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:28.600145 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.594212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.0.1:35494)
0620 14:19:28.595281 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.595291 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.595337 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.595614 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.595615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.595617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:28.595650 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.595651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.597311 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.597312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.598396 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.598398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.599783 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.599784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.599988 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.599992 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.600045 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.600079 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":68,"threads_started":1}
I20260620 14:19:28.600344 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.594398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.0.98.1:35681)
0620 14:19:28.594687 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.594692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.595393 (+ 701us) server_negotiation.cc:407] Connection header received
0620 14:19:28.595436 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.595439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.595455 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.595521 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:28.595723 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.597228 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.598482 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.599627 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.599685 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.599749 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.600232 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.600270 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":77,"threads_started":1}
I20260620 14:19:28.600345 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c34198e4244194a1db3d8b96639fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:28.600522 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: attempting to promote NON_VOTER 68ac82958a264fdda67bdc8eb238c2f0 to VOTER
I20260620 14:19:28.600494 7658 tablet_service.cc:1467] Processing CreateTablet for tablet b8c34198e4244194a1db3d8b96639fa3 (DEFAULT_TABLE table=test-table [id=590eab83c2cf4c7aa82e5c87ce681398]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:28.600638 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:28.600837 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c34198e4244194a1db3d8b96639fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:28.601555 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Refusing update from remote peer 10c59e1e1ea54a74ad3fc0f3f761c456: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260620 14:19:28.601780 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260620 14:19:28.602346 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.603236 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.603302 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:28.603349 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.603391 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:28.603691 7754 tablet_bootstrap.cc:492] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Bootstrap starting.
I20260620 14:19:28.603649 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.604306 7754 tablet_bootstrap.cc:654] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:28.604384 7385 tablet_service.cc:1467] Processing CreateTablet for tablet b8c34198e4244194a1db3d8b96639fa3 (DEFAULT_TABLE table=test-table [id=590eab83c2cf4c7aa82e5c87ce681398]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:28.604583 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.604678 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:19:28.604635 7754 log.cc:824] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:28.604714 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c34198e4244194a1db3d8b96639fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:28.604883 7275 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 LEARNER]: Refusing update from remote peer 10c59e1e1ea54a74ad3fc0f3f761c456: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260620 14:19:28.605093 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260620 14:19:28.605473 7275 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.606237 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.606335 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:19:28.607636 7756 tablet_bootstrap.cc:492] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Bootstrap starting.
I20260620 14:19:28.608144 7756 tablet_bootstrap.cc:654] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:28.608384 7757 tablet_bootstrap.cc:492] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Bootstrap starting.
I20260620 14:19:28.608451 7756 log.cc:824] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:28.608901 7754 tablet_bootstrap.cc:492] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: No bootstrap required, opened a new log
I20260620 14:19:28.608944 7757 tablet_bootstrap.cc:654] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:28.608992 7754 ts_tablet_manager.cc:1397] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 14:19:28.609213 7757 log.cc:824] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:28.609855 7757 tablet_bootstrap.cc:492] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: No bootstrap required, opened a new log
I20260620 14:19:28.609906 7757 ts_tablet_manager.cc:1397] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:28.611521 7754 raft_consensus.cc:348] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.611726 7754 raft_consensus.cc:374] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:28.611774 7754 raft_consensus.cc:729] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b0c869b06c4f3fac1d420969f912a6, State: Initialized, Role: FOLLOWER
I20260620 14:19:28.611696 7757 raft_consensus.cc:348] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.611858 7757 raft_consensus.cc:374] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:28.611970 7757 raft_consensus.cc:729] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a063836e372f4021b8f5f7d65ff18c6b, State: Initialized, Role: FOLLOWER
I20260620 14:19:28.611965 7754 consensus_queue.cc:260] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.612084 7757 consensus_queue.cc:260] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.612233 7754 ts_tablet_manager.cc:1428] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:19:28.612320 7725 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
I20260620 14:19:28.612352 7757 ts_tablet_manager.cc:1428] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:19:28.612640 7589 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
I20260620 14:19:28.612988 7756 tablet_bootstrap.cc:492] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: No bootstrap required, opened a new log
I20260620 14:19:28.613051 7756 ts_tablet_manager.cc:1397] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 14:19:28.615494 7756 raft_consensus.cc:348] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.615664 7756 raft_consensus.cc:374] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:28.615722 7756 raft_consensus.cc:729] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf0c092f8ac435da3112368a2180822, State: Initialized, Role: FOLLOWER
I20260620 14:19:28.615849 7756 consensus_queue.cc:260] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.616127 7756 ts_tablet_manager.cc:1428] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:19:28.616182 7452 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
W20260620 14:19:28.680162 7591 tablet.cc:2369] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:19:28.801061 7454 tablet.cc:2369] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:19:28.816313 7727 tablet.cc:2369] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:28.958859 7760 raft_consensus.cc:482] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:28.959020 7760 raft_consensus.cc:504] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.959438 7760 leader_election.cc:288] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2:39859), acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681)
I20260620 14:19:28.962929 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.959689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60455 (local address 127.0.98.2:39859)
0620 14:19:28.959846 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.959848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.959857 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:28.959886 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.959887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.959901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.959950 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:28.960152 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.960995 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.962191 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.962559 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.962651 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.962662 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.962828 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.962866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260620 14:19:28.962929 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.959637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39859 (local address 127.0.98.3:60455)
0620 14:19:28.959695 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.959709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.959753 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.960000 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.960001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.960003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:28.960079 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.960081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.961081 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.961082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.962108 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.962110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.962652 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.962653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.962732 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.962735 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.962815 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.962858 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:19:28.963238 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.959752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.98.3:45817)
0620 14:19:28.959847 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:28.959857 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:28.959941 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:28.960091 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:28.960091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:28.960093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:28.960155 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.960156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.960815 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.960816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.961964 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:28.961966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.962912 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.962913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:28.963019 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.963023 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:28.963075 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:19:28.963127 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:19:28.963299 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34198e4244194a1db3d8b96639fa3" candidate_uuid: "d8b0c869b06c4f3fac1d420969f912a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a063836e372f4021b8f5f7d65ff18c6b" is_pre_election: true
I20260620 14:19:28.963456 7542 raft_consensus.cc:2454] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b0c869b06c4f3fac1d420969f912a6 in term 0.
I20260620 14:19:28.963486 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:28.959706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45817 (local address 127.0.98.1:35681)
0620 14:19:28.959749 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:28.959751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:28.959927 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:19:28.959997 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:28.959999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:28.960012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:28.960088 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:28.960231 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.960704 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.962071 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:28.962786 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:28.962865 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:28.962876 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:28.963339 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 14:19:28.963393 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:19:28.963791 7614 leader_election.cc:302] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a063836e372f4021b8f5f7d65ff18c6b, d8b0c869b06c4f3fac1d420969f912a6; no voters:
I20260620 14:19:28.963941 7760 raft_consensus.cc:2781] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:28.963929 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34198e4244194a1db3d8b96639fa3" candidate_uuid: "d8b0c869b06c4f3fac1d420969f912a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf0c092f8ac435da3112368a2180822" is_pre_election: true
I20260620 14:19:28.964020 7760 raft_consensus.cc:482] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:28.964052 7760 raft_consensus.cc:3037] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:28.964084 7405 raft_consensus.cc:2454] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b0c869b06c4f3fac1d420969f912a6 in term 0.
I20260620 14:19:28.964812 7760 raft_consensus.cc:504] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.964958 7760 leader_election.cc:288] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [CANDIDATE]: Term 1 election: Requested vote from peers a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2:39859), acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681)
I20260620 14:19:28.965086 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34198e4244194a1db3d8b96639fa3" candidate_uuid: "d8b0c869b06c4f3fac1d420969f912a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a063836e372f4021b8f5f7d65ff18c6b"
I20260620 14:19:28.965179 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34198e4244194a1db3d8b96639fa3" candidate_uuid: "d8b0c869b06c4f3fac1d420969f912a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf0c092f8ac435da3112368a2180822"
I20260620 14:19:28.965188 7542 raft_consensus.cc:3037] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:28.965255 7405 raft_consensus.cc:3037] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:28.966069 7405 raft_consensus.cc:2454] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b0c869b06c4f3fac1d420969f912a6 in term 1.
I20260620 14:19:28.966065 7542 raft_consensus.cc:2454] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b0c869b06c4f3fac1d420969f912a6 in term 1.
I20260620 14:19:28.966246 7614 leader_election.cc:302] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a063836e372f4021b8f5f7d65ff18c6b, d8b0c869b06c4f3fac1d420969f912a6; no voters:
I20260620 14:19:28.966333 7760 raft_consensus.cc:2781] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:28.966468 7760 raft_consensus.cc:686] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 1 LEADER]: Becoming Leader. State: Replica: d8b0c869b06c4f3fac1d420969f912a6, State: Running, Role: LEADER
I20260620 14:19:28.966557 7760 consensus_queue.cc:237] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260620 14:19:28.967376 6489 catalog_manager.cc:5495] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d8b0c869b06c4f3fac1d420969f912a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } health_report { overall_health: UNKNOWN } } }
I20260620 14:19:28.968139 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:28.968250 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:29.043486 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:29.041273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.0.98.3:39473)
0620 14:19:29.041315 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:29.041317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:29.041332 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:19:29.041406 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:29.041407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:29.041428 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:29.041485 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:29.041599 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.042071 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.042775 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.042975 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.043073 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:29.043195 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:29.043292 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:29.043352 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:29.043388 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 14:19:29.043404 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:19:29.045322 7405 raft_consensus.cc:1261] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Refusing update from remote peer d8b0c869b06c4f3fac1d420969f912a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:29.045322 7542 raft_consensus.cc:1261] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 1 FOLLOWER]: Refusing update from remote peer d8b0c869b06c4f3fac1d420969f912a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:29.045727 7763 consensus_queue.cc:1035] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:29.045856 7760 consensus_queue.cc:1035] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:29.048092 7766 mvcc.cc:205] Tried to move back new op lower bound from 7298929332406796288 to 7298929332087357440. Current Snapshot: MvccSnapshot[applied={T|T < 7298929332406796288}]
I20260620 14:19:29.048722 7768 mvcc.cc:205] Tried to move back new op lower bound from 7298929332406796288 to 7298929332087357440. Current Snapshot: MvccSnapshot[applied={T|T < 7298929332406796288}]
I20260620 14:19:29.049553 7767 mvcc.cc:205] Tried to move back new op lower bound from 7298929332406796288 to 7298929332087357440. Current Snapshot: MvccSnapshot[applied={T|T < 7298929332406796288}]
I20260620 14:19:29.051625 6488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:29.051729 6488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:29.053048 6488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10c59e1e1ea54a74ad3fc0f3f761c456: Sending DeleteTablet for 3 replicas of tablet b8c34198e4244194a1db3d8b96639fa3
I20260620 14:19:29.053355 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c34198e4244194a1db3d8b96639fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35494
I20260620 14:19:29.053357 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c34198e4244194a1db3d8b96639fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54422
I20260620 14:19:29.053629 7776 tablet_replica.cc:330] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: stopping tablet replica
I20260620 14:19:29.053617 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c34198e4244194a1db3d8b96639fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44606
I20260620 14:19:29.053725 7776 raft_consensus.cc:2229] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:29.053819 7776 raft_consensus.cc:2258] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:29.053964 7777 tablet_replica.cc:330] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: stopping tablet replica
I20260620 14:19:29.054054 7777 raft_consensus.cc:2229] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:29.054147 7777 raft_consensus.cc:2258] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:29.054613 7776 ts_tablet_manager.cc:1905] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:29.054679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6457
I20260620 14:19:29.055482 7777 ts_tablet_manager.cc:1905] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:29.056258 7776 ts_tablet_manager.cc:1918] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:29.056336 7776 log.cc:1196] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/b8c34198e4244194a1db3d8b96639fa3
I20260620 14:19:29.056610 7776 ts_tablet_manager.cc:1939] T b8c34198e4244194a1db3d8b96639fa3 P acf0c092f8ac435da3112368a2180822: Deleting consensus metadata
I20260620 14:19:29.057047 7777 ts_tablet_manager.cc:1918] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:29.057102 7777 log.cc:1196] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/b8c34198e4244194a1db3d8b96639fa3
I20260620 14:19:29.057283 7777 ts_tablet_manager.cc:1939] T b8c34198e4244194a1db3d8b96639fa3 P a063836e372f4021b8f5f7d65ff18c6b: Deleting consensus metadata
W20260620 14:19:29.060087 7341 connection.cc:535] server connection from 127.0.0.1:35494 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:19:29.060102 7478 connection.cc:535] server connection from 127.0.0.1:54422 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:19:29.060266 7778 tablet_replica.cc:330] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: stopping tablet replica
I20260620 14:19:29.060429 7778 raft_consensus.cc:2229] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:29.060618 7778 raft_consensus.cc:2258] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:29.061141 7778 ts_tablet_manager.cc:1905] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:29.063017 7778 ts_tablet_manager.cc:1918] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:29.063145 7778 log.cc:1196] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/b8c34198e4244194a1db3d8b96639fa3
I20260620 14:19:29.063500 7778 ts_tablet_manager.cc:1939] T b8c34198e4244194a1db3d8b96639fa3 P d8b0c869b06c4f3fac1d420969f912a6: Deleting consensus metadata
W20260620 14:19:29.063916 7611 connection.cc:441] server connection from 127.0.0.1:44606 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44606 (request call id 1) could send its response
I20260620 14:19:29.068122 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:29.054163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.0.98.60:38525)
0620 14:19:29.054205 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:29.054207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:29.054733 (+ 526us) server_negotiation.cc:407] Connection header received
0620 14:19:29.060229 (+ 5496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:29.060231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:29.060244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:29.060309 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:29.061087 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.061889 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.063056 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.063341 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.063382 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:29.065067 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:29.065069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:29.065069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:29.065077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:29.066817 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:29.067255 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:29.067256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:29.067257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:29.067311 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:29.067520 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:29.067521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:29.067521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:29.067664 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:29.067736 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:29.067929 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:19:29.068007 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:19:29.071158 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:29.062650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.0.98.61:33293)
0620 14:19:29.062690 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:29.062692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:29.063254 (+ 562us) server_negotiation.cc:407] Connection header received
0620 14:19:29.063352 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:29.063353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:29.063385 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:29.063416 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:29.064210 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.064987 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.066003 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.066303 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.066340 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:29.067932 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:29.067934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:29.067934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:29.067940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:29.069966 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:29.070338 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:29.070339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:29.070340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:29.070398 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:29.070612 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:29.070612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:29.070613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:29.070728 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:29.070790 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:29.070972 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 14:19:29.071054 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:19:29.071712 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:29.062910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.0.98.60:38525)
0620 14:19:29.062962 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:29.062964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:29.064278 (+ 1314us) server_negotiation.cc:407] Connection header received
0620 14:19:29.064363 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:29.064363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:29.064372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:29.064400 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:29.065000 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.065720 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.066717 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:29.067039 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:29.067084 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:29.068641 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:29.068642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:29.068643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:29.068650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:29.070334 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:29.070640 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:29.070641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:29.070641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:29.070690 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:29.070891 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:29.070892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:29.070892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:29.071056 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:29.071143 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:29.071519 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 14:19:29.071619 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:19:30.049757 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.049388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:52389)
0620 14:19:30.049589 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.049687 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260620 14:19:30.050015 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.049679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:35553)
0620 14:19:30.049899 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.049965 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":57,"threads_started":1}
W20260620 14:19:30.050202 7589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43329 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
W20260620 14:19:30.050261 7725 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43329 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.050513 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.049679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:40165)
0620 14:19:30.050403 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.050470 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":48,"threads_started":1}
W20260620 14:19:30.050710 7452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43329 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.553848 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10c59e1e1ea54a74ad3fc0f3f761c456)
I20260620 14:19:30.553936 7791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10c59e1e1ea54a74ad3fc0f3f761c456)
I20260620 14:19:30.554008 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:30.554085 7791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:30.554342 7791 leader_election.cc:288] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329), 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60:38525)
I20260620 14:19:30.554378 7790 leader_election.cc:288] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329), 04049bf957fc4fa4836fb4abe2149c97 (127.0.98.61:33293)
I20260620 14:19:30.554878 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.554519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57972)
0620 14:19:30.554767 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.554827 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":58,"threads_started":1}
W20260620 14:19:30.555007 7071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.555150 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.554519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57970)
0620 14:19:30.554848 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.554895 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":50,"threads_started":1}
W20260620 14:19:30.555339 7256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.555403 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.555315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57980)
0620 14:19:30.555349 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.555369 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:19:30.555491 7071 leader_election.cc:334] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.555816 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.555739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57982)
0620 14:19:30.555772 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.555793 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:19:30.555886 7256 leader_election.cc:334] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.557893 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.554671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33293 (local address 127.0.0.1:33324)
0620 14:19:30.554932 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.554941 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:30.555003 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:30.555221 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:30.555222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:30.555224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:30.555275 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:30.555277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.556225 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.556226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:30.557034 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:30.557036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.557674 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.557675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:30.557731 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:30.557734 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:30.557783 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:19:30.557812 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":41,"threads_started":1}
I20260620 14:19:30.557904 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.554723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.98.61:33293)
0620 14:19:30.554899 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:30.554902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:30.555062 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:19:30.555087 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:30.555088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:30.555098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:30.555165 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:30.555349 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.556131 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.557118 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.557598 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.557652 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:30.557661 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:30.557801 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:19:30.557841 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":49,"threads_started":1}
I20260620 14:19:30.557996 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.554805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.0.98.60:38525)
0620 14:19:30.554982 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:30.554984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:30.554990 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:19:30.555016 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:30.555017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:30.555029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:30.555268 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:30.555303 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.556082 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.557016 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.557613 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.557702 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:30.557712 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:30.557882 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:19:30.557928 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":51,"threads_started":1}
I20260620 14:19:30.558193 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.554678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38525 (local address 127.0.0.1:38896)
0620 14:19:30.554807 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.554819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:30.554873 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:30.555115 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:30.555116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:30.555118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:30.555199 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:30.555202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.556169 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.556170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:30.556922 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:30.556923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:30.557712 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:30.557712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:30.557801 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:30.557805 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:30.557866 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:19:30.557900 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":37,"threads_started":1}
I20260620 14:19:30.558264 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68ac82958a264fdda67bdc8eb238c2f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04049bf957fc4fa4836fb4abe2149c97" is_pre_election: true
I20260620 14:19:30.558355 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ac82958a264fdda67bdc8eb238c2f0 in term 1.
I20260620 14:19:30.558422 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04049bf957fc4fa4836fb4abe2149c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68ac82958a264fdda67bdc8eb238c2f0" is_pre_election: true
I20260620 14:19:30.558496 7256 leader_election.cc:302] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04049bf957fc4fa4836fb4abe2149c97, 68ac82958a264fdda67bdc8eb238c2f0; no voters: 10c59e1e1ea54a74ad3fc0f3f761c456
I20260620 14:19:30.558522 7275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04049bf957fc4fa4836fb4abe2149c97 in term 1.
I20260620 14:19:30.558586 7790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:19:30.558616 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10c59e1e1ea54a74ad3fc0f3f761c456)
I20260620 14:19:30.558640 7790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:19:30.558656 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04049bf957fc4fa4836fb4abe2149c97, 68ac82958a264fdda67bdc8eb238c2f0; no voters: 10c59e1e1ea54a74ad3fc0f3f761c456
I20260620 14:19:30.558753 7791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:19:30.558785 7791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10c59e1e1ea54a74ad3fc0f3f761c456)
I20260620 14:19:30.558811 7791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:19:30.559334 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:30.559448 7790 leader_election.cc:288] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [CANDIDATE]: Term 2 election: Requested vote from peers 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329), 04049bf957fc4fa4836fb4abe2149c97 (127.0.98.61:33293)
I20260620 14:19:30.559466 7791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:30.559681 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.559523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:57992)
0620 14:19:30.559631 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.559655 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 14:19:30.559664 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68ac82958a264fdda67bdc8eb238c2f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04049bf957fc4fa4836fb4abe2149c97"
I20260620 14:19:30.559746 7097 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 68ac82958a264fdda67bdc8eb238c2f0 in current term 2: Already voted for candidate 04049bf957fc4fa4836fb4abe2149c97 in this term.
I20260620 14:19:30.559799 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.559665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58006)
0620 14:19:30.559722 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.559749 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 14:19:30.559819 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04049bf957fc4fa4836fb4abe2149c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68ac82958a264fdda67bdc8eb238c2f0"
I20260620 14:19:30.559899 7275 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 04049bf957fc4fa4836fb4abe2149c97 in current term 2: Already voted for candidate 68ac82958a264fdda67bdc8eb238c2f0 in this term.
I20260620 14:19:30.559942 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.559854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58010)
0620 14:19:30.559887 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.559914 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 14:19:30.559991 7256 leader_election.cc:334] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.560034 7256 leader_election.cc:302] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68ac82958a264fdda67bdc8eb238c2f0; no voters: 04049bf957fc4fa4836fb4abe2149c97, 10c59e1e1ea54a74ad3fc0f3f761c456
I20260620 14:19:30.560089 7790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:19:30.560226 7791 leader_election.cc:288] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 2 election: Requested vote from peers 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329), 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60:38525)
I20260620 14:19:30.560279 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:30.560186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58014)
0620 14:19:30.560215 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:30.560239 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 14:19:30.560326 7071 leader_election.cc:334] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:30.560379 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04049bf957fc4fa4836fb4abe2149c97; no voters: 10c59e1e1ea54a74ad3fc0f3f761c456, 68ac82958a264fdda67bdc8eb238c2f0
I20260620 14:19:30.560448 7791 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:19:31.050884 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:31.050602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:59285)
0620 14:19:31.050780 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:31.050824 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260620 14:19:31.050931 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:31.050622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:36861)
0620 14:19:31.050781 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:31.050883 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260620 14:19:31.051505 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:31.051252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:33137)
0620 14:19:31.051412 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:31.051457 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":59,"threads_started":1}
I20260620 14:19:32.052033 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.051714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:56451)
0620 14:19:32.051920 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.051963 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
I20260620 14:19:32.052150 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.051716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:60067)
0620 14:19:32.052052 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.052091 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":95,"threads_started":1}
I20260620 14:19:32.052541 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.052034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:55589)
0620 14:19:32.052461 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.052496 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":45,"threads_started":1}
I20260620 14:19:32.388021 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:32.388124 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:32.388296 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329), 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60:38525)
I20260620 14:19:32.388688 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.388455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58294)
0620 14:19:32.388583 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.388633 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260620 14:19:32.388763 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04049bf957fc4fa4836fb4abe2149c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68ac82958a264fdda67bdc8eb238c2f0" is_pre_election: true
I20260620 14:19:32.388899 7275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04049bf957fc4fa4836fb4abe2149c97 in term 2.
I20260620 14:19:32.389119 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.388991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58308)
0620 14:19:32.389037 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.389088 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:19:32.389125 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04049bf957fc4fa4836fb4abe2149c97, 68ac82958a264fdda67bdc8eb238c2f0; no voters:
W20260620 14:19:32.389228 7071 leader_election.cc:334] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:32.389231 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 14:19:32.389278 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:32.389307 7806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:19:32.390019 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:32.390161 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 3 election: Requested vote from peers 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329), 68ac82958a264fdda67bdc8eb238c2f0 (127.0.98.60:38525)
I20260620 14:19:32.390332 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.390250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58316)
0620 14:19:32.390281 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.390307 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 14:19:32.390429 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04049bf957fc4fa4836fb4abe2149c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "68ac82958a264fdda67bdc8eb238c2f0"
I20260620 14:19:32.390530 7275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:19:32.390693 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.390616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58332)
0620 14:19:32.390646 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.390668 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 14:19:32.390784 7071 leader_election.cc:334] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260620 14:19:32.391433 7275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04049bf957fc4fa4836fb4abe2149c97 in term 3.
I20260620 14:19:32.391628 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04049bf957fc4fa4836fb4abe2149c97, 68ac82958a264fdda67bdc8eb238c2f0; no voters: 10c59e1e1ea54a74ad3fc0f3f761c456
I20260620 14:19:32.391731 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:19:32.391844 7806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [term 3 LEADER]: Becoming Leader. State: Replica: 04049bf957fc4fa4836fb4abe2149c97, State: Running, Role: LEADER
I20260620 14:19:32.391919 7806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } }
I20260620 14:19:32.392309 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04049bf957fc4fa4836fb4abe2149c97. Latest consensus state: current_term: 3 leader_uuid: "04049bf957fc4fa4836fb4abe2149c97" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:32.392411 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:32.392575 7811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:32.392961 7811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:32.393013 7811 catalog_manager.cc:1259] Loaded cluster ID: eb1a048fae624342b5aa6e9d54fc02fa
I20260620 14:19:32.393028 7811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:32.393159 7811 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:32.393208 7811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Loaded TSK: 0
I20260620 14:19:32.393306 7811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:32.477820 7085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:32.479493 7085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:32.480525 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.480396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58354)
0620 14:19:32.480460 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.480492 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:19:32.480659 7275 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [term 3 FOLLOWER]: Refusing update from remote peer 04049bf957fc4fa4836fb4abe2149c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 14:19:32.480895 7806 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:19:32.480960 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.480801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.0.1:58370)
0620 14:19:32.480855 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.480885 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:19:32.481134 7071 consensus_peers.cc:487] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 -> Peer 10c59e1e1ea54a74ad3fc0f3f761c456 (127.0.98.62:43329): Couldn't send request to peer 10c59e1e1ea54a74ad3fc0f3f761c456. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:19:32.481644 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04049bf957fc4fa4836fb4abe2149c97. Latest consensus state: current_term: 3 leader_uuid: "04049bf957fc4fa4836fb4abe2149c97" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:32.481756 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:19:32.482172 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "04049bf957fc4fa4836fb4abe2149c97" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:32.482254 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:32.482331 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "04049bf957fc4fa4836fb4abe2149c97" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "10c59e1e1ea54a74ad3fc0f3f761c456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } peers { permanent_uuid: "04049bf957fc4fa4836fb4abe2149c97" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33293 } attrs { promote: false } } peers { permanent_uuid: "68ac82958a264fdda67bdc8eb238c2f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38525 } attrs { promote: false } } }
I20260620 14:19:32.482466 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ac82958a264fdda67bdc8eb238c2f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:19:32.486969 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.483924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39473 (local address 127.0.0.1:44626)
0620 14:19:32.484184 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.484193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:32.484244 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:32.484500 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:32.484501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:32.484502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:32.484550 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.484551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.485158 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.485159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.485972 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.485974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.486712 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.486713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.486776 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.486779 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:32.486861 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:19:32.486900 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":74,"threads_started":1}
I20260620 14:19:32.486969 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.483987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.0.98.3:39473)
0620 14:19:32.484327 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:32.484329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:32.484336 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:32.484368 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:32.484369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:32.484377 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:32.484498 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:32.484609 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.485079 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.486133 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.486616 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.486711 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.486723 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:32.486868 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:19:32.486904 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":64,"threads_started":1}
I20260620 14:19:32.487185 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.483896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.0.1:35498)
0620 14:19:32.484053 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.484064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:32.484114 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:32.484335 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:32.484336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:32.484337 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:32.484440 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.484442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.485066 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.485068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.486228 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.486229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.486982 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.486983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.487040 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.487044 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:32.487087 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:19:32.487122 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 14:19:32.487357 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.483957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.0.98.1:35681)
0620 14:19:32.484112 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:32.484114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:32.484124 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:32.484182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:32.484183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:32.484194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:32.484252 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:32.484495 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.484982 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.486311 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.486875 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.486942 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.486952 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:32.487233 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 14:19:32.487269 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
I20260620 14:19:32.487588 7658 tablet_service.cc:1467] Processing CreateTablet for tablet e9c2795042b7483199b73736de2c143b (DEFAULT_TABLE table=test-table [id=fc64f26ccca54f9097fd75c38073a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:32.487727 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c2795042b7483199b73736de2c143b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:32.488363 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.484115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.0.98.2:39859)
0620 14:19:32.484276 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:32.484277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:32.485234 (+ 957us) server_negotiation.cc:407] Connection header received
0620 14:19:32.485266 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:32.485267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:32.485278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:32.485336 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:32.485430 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.486141 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.487370 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.487988 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.488028 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.488036 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:32.488205 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:19:32.488259 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":49,"threads_started":1}
I20260620 14:19:32.488241 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.484053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39859 (local address 127.0.0.1:54424)
0620 14:19:32.484568 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.484577 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:32.485220 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:32.485344 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:32.485345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:32.485346 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:32.485379 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.485380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.486228 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.486229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.487281 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.487283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.488081 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.488082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.488131 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.488133 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:32.488172 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 14:19:32.488198 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":67,"threads_started":1}
I20260620 14:19:32.489081 7823 tablet_bootstrap.cc:492] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: Bootstrap starting.
I20260620 14:19:32.489265 7385 tablet_service.cc:1467] Processing CreateTablet for tablet e9c2795042b7483199b73736de2c143b (DEFAULT_TABLE table=test-table [id=fc64f26ccca54f9097fd75c38073a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:32.489387 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c2795042b7483199b73736de2c143b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:32.489420 7522 tablet_service.cc:1467] Processing CreateTablet for tablet e9c2795042b7483199b73736de2c143b (DEFAULT_TABLE table=test-table [id=fc64f26ccca54f9097fd75c38073a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:32.489521 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c2795042b7483199b73736de2c143b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:32.489581 7823 tablet_bootstrap.cc:654] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:32.490111 7823 tablet_bootstrap.cc:492] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: No bootstrap required, opened a new log
I20260620 14:19:32.490150 7823 ts_tablet_manager.cc:1397] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:32.490232 7823 raft_consensus.cc:348] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.490278 7823 raft_consensus.cc:374] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:32.490291 7823 raft_consensus.cc:729] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b0c869b06c4f3fac1d420969f912a6, State: Initialized, Role: FOLLOWER
I20260620 14:19:32.490352 7823 consensus_queue.cc:260] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.490582 7823 ts_tablet_manager.cc:1428] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:32.490804 7826 tablet_bootstrap.cc:492] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822: Bootstrap starting.
I20260620 14:19:32.490837 7827 tablet_bootstrap.cc:492] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: Bootstrap starting.
I20260620 14:19:32.491132 7826 tablet_bootstrap.cc:654] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:32.491192 7827 tablet_bootstrap.cc:654] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:32.491217 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.491005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:59123)
0620 14:19:32.491158 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.491189 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 14:19:32.491823 7827 tablet_bootstrap.cc:492] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: No bootstrap required, opened a new log
I20260620 14:19:32.491884 7827 ts_tablet_manager.cc:1397] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:32.491914 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.491757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:50631)
0620 14:19:32.491849 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.491883 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:19:32.492095 7827 raft_consensus.cc:348] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.492126 7826 tablet_bootstrap.cc:492] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822: No bootstrap required, opened a new log
I20260620 14:19:32.492177 7827 raft_consensus.cc:374] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:32.492204 7827 raft_consensus.cc:729] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a063836e372f4021b8f5f7d65ff18c6b, State: Initialized, Role: FOLLOWER
I20260620 14:19:32.492185 7826 ts_tablet_manager.cc:1397] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:32.492252 7827 consensus_queue.cc:260] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.492347 7826 raft_consensus.cc:348] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.492429 7826 raft_consensus.cc:374] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:32.492405 7827 ts_tablet_manager.cc:1428] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:32.492450 7826 raft_consensus.cc:729] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf0c092f8ac435da3112368a2180822, State: Initialized, Role: FOLLOWER
I20260620 14:19:32.492489 7826 consensus_queue.cc:260] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.492684 7826 ts_tablet_manager.cc:1428] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:32.492703 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.492615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:53981)
0620 14:19:32.492650 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.492680 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:19:32.493117 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.493002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:40155)
0620 14:19:32.493058 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.493089 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:19:32.493424 7453 heartbeater.cc:503] Master 127.0.98.61:33293 was elected leader, sending a full tablet report...
I20260620 14:19:32.493494 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.493362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:40549)
0620 14:19:32.493411 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.493439 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:19:32.493705 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.493593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:35927)
0620 14:19:32.493652 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.493677 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:19:32.493932 7590 heartbeater.cc:503] Master 127.0.98.61:33293 was elected leader, sending a full tablet report...
I20260620 14:19:32.603825 7830 raft_consensus.cc:482] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:32.603958 7830 raft_consensus.cc:504] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.604332 7830 leader_election.cc:288] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681), d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3:39473)
I20260620 14:19:32.607456 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.604559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39473 (local address 127.0.98.2:47319)
0620 14:19:32.604656 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.604670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:32.604763 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:32.604926 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:32.604927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:32.604942 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:32.605016 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.605017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.605688 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.605689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.606507 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.606509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.607247 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.607248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.607316 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.607320 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:32.607367 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:19:32.607398 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 14:19:32.607503 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.604688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47319 (local address 127.0.98.3:39473)
0620 14:19:32.604753 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:32.604755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:32.604766 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:32.604836 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:32.604837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:32.604851 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:32.604924 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:32.605112 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.605616 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.606591 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.607155 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.607229 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.607242 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:32.607394 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 14:19:32.607433 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:32.607771 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2795042b7483199b73736de2c143b" candidate_uuid: "a063836e372f4021b8f5f7d65ff18c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b0c869b06c4f3fac1d420969f912a6" is_pre_election: true
I20260620 14:19:32.607901 7678 raft_consensus.cc:2454] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a063836e372f4021b8f5f7d65ff18c6b in term 0.
I20260620 14:19:32.608104 7477 leader_election.cc:302] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a063836e372f4021b8f5f7d65ff18c6b, d8b0c869b06c4f3fac1d420969f912a6; no voters:
I20260620 14:19:32.608282 7830 raft_consensus.cc:2781] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:32.608337 7830 raft_consensus.cc:482] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:32.608369 7830 raft_consensus.cc:3037] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:32.608687 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.604639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39103 (local address 127.0.98.1:35681)
0620 14:19:32.604714 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:32.604717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:32.604995 (+ 278us) server_negotiation.cc:407] Connection header received
0620 14:19:32.605066 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:32.605067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:32.605079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:32.605130 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:32.605846 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.606310 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.607734 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.608353 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.608420 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.608431 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:32.608607 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:19:32.608641 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:19:32.608709 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.604564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.98.2:39103)
0620 14:19:32.604893 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.604903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:32.605005 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:32.605697 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:32.605698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:32.605699 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:32.605753 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.605754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.606398 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.606399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.607493 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:32.607495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.608442 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.608443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:32.608519 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.608525 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:19:32.608595 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:19:32.608635 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":154,"thread_start_us":125,"threads_started":1}
I20260620 14:19:32.608946 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2795042b7483199b73736de2c143b" candidate_uuid: "a063836e372f4021b8f5f7d65ff18c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf0c092f8ac435da3112368a2180822" is_pre_election: true
I20260620 14:19:32.609037 7405 raft_consensus.cc:2454] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a063836e372f4021b8f5f7d65ff18c6b in term 0.
I20260620 14:19:32.609462 7830 raft_consensus.cc:504] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.609607 7830 leader_election.cc:288] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [CANDIDATE]: Term 1 election: Requested vote from peers acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681), d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3:39473)
I20260620 14:19:32.609730 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2795042b7483199b73736de2c143b" candidate_uuid: "a063836e372f4021b8f5f7d65ff18c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf0c092f8ac435da3112368a2180822"
I20260620 14:19:32.609818 7405 raft_consensus.cc:3037] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:32.609805 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c2795042b7483199b73736de2c143b" candidate_uuid: "a063836e372f4021b8f5f7d65ff18c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b0c869b06c4f3fac1d420969f912a6"
I20260620 14:19:32.609957 7678 raft_consensus.cc:3037] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:32.610427 7405 raft_consensus.cc:2454] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a063836e372f4021b8f5f7d65ff18c6b in term 1.
I20260620 14:19:32.610638 7476 leader_election.cc:302] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a063836e372f4021b8f5f7d65ff18c6b, acf0c092f8ac435da3112368a2180822; no voters:
I20260620 14:19:32.610726 7830 raft_consensus.cc:2781] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:32.610719 7678 raft_consensus.cc:2454] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a063836e372f4021b8f5f7d65ff18c6b in term 1.
I20260620 14:19:32.610870 7830 raft_consensus.cc:686] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 1 LEADER]: Becoming Leader. State: Replica: a063836e372f4021b8f5f7d65ff18c6b, State: Running, Role: LEADER
I20260620 14:19:32.610984 7830 consensus_queue.cc:237] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } }
I20260620 14:19:32.611203 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.611098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:43623)
0620 14:19:32.611135 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.611163 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:19:32.611752 7085 catalog_manager.cc:5495] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a063836e372f4021b8f5f7d65ff18c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a063836e372f4021b8f5f7d65ff18c6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39859 } health_report { overall_health: HEALTHY } } }
I20260620 14:19:32.631446 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.629266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.98.2:39859)
0620 14:19:32.629329 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:32.629331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:32.629354 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:19:32.629398 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:32.629399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:32.629416 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:32.629477 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:32.629607 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.630069 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.630753 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:32.630940 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:32.630994 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:32.631130 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:32.631224 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:32.631275 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:32.631314 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:19:32.631331 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:32.632685 7405 raft_consensus.cc:1261] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Refusing update from remote peer a063836e372f4021b8f5f7d65ff18c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:32.632697 7678 raft_consensus.cc:1261] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 1 FOLLOWER]: Refusing update from remote peer a063836e372f4021b8f5f7d65ff18c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:32.632928 7830 consensus_queue.cc:1035] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "acf0c092f8ac435da3112368a2180822" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:32.633073 7833 consensus_queue.cc:1035] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b0c869b06c4f3fac1d420969f912a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:32.633909 7726 heartbeater.cc:503] Master 127.0.98.61:33293 was elected leader, sending a full tablet report...
I20260620 14:19:32.634143 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.633987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:36193)
0620 14:19:32.634079 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.634110 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:19:32.634164 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.633997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:57365)
0620 14:19:32.634101 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.634132 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 14:19:32.635478 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.635243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:49027)
0620 14:19:32.635410 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.635445 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 14:19:32.636191 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.636078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:56253)
0620 14:19:32.636136 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.636164 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:19:32.636592 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.636488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:35489)
0620 14:19:32.636526 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.636560 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:19:32.636945 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:32.636820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:50017)
0620 14:19:32.636879 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.636904 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:19:32.637665 7085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:32.637739 7085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:32.638927 7085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04049bf957fc4fa4836fb4abe2149c97: Sending DeleteTablet for 3 replicas of tablet e9c2795042b7483199b73736de2c143b
I20260620 14:19:32.639297 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c2795042b7483199b73736de2c143b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:32 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54424
I20260620 14:19:32.639297 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c2795042b7483199b73736de2c143b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:32 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35498
I20260620 14:19:32.639642 7847 tablet_replica.cc:330] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822: stopping tablet replica
I20260620 14:19:32.639659 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c2795042b7483199b73736de2c143b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:32 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44626
I20260620 14:19:32.639703 7847 raft_consensus.cc:2229] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:32.639747 7847 raft_consensus.cc:2258] T e9c2795042b7483199b73736de2c143b P acf0c092f8ac435da3112368a2180822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:32.639816 7848 tablet_replica.cc:330] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: stopping tablet replica
I20260620 14:19:32.639863 7848 raft_consensus.cc:2229] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:32.639896 7848 raft_consensus.cc:2258] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:32.640014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 7234
I20260620 14:19:32.640170 7848 ts_tablet_manager.cc:1905] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:32.640576 7846 tablet_replica.cc:330] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: stopping tablet replica
I20260620 14:19:32.640627 7846 raft_consensus.cc:2229] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:32.640700 7846 raft_consensus.cc:2258] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:32.642134 7848 ts_tablet_manager.cc:1918] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:32.642186 7848 log.cc:1196] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/e9c2795042b7483199b73736de2c143b
I20260620 14:19:32.642419 7848 ts_tablet_manager.cc:1939] T e9c2795042b7483199b73736de2c143b P d8b0c869b06c4f3fac1d420969f912a6: Deleting consensus metadata
I20260620 14:19:32.642772 7073 catalog_manager.cc:4841] TS d8b0c869b06c4f3fac1d420969f912a6 (127.0.98.3:39473): tablet e9c2795042b7483199b73736de2c143b (table test-table [id=fc64f26ccca54f9097fd75c38073a701]) successfully deleted
I20260620 14:19:32.643779 7846 ts_tablet_manager.cc:1905] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:32.645359 7846 ts_tablet_manager.cc:1918] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:32.645421 7846 log.cc:1196] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/e9c2795042b7483199b73736de2c143b
I20260620 14:19:32.645645 7846 ts_tablet_manager.cc:1939] T e9c2795042b7483199b73736de2c143b P a063836e372f4021b8f5f7d65ff18c6b: Deleting consensus metadata
I20260620 14:19:32.645977 7074 catalog_manager.cc:4841] TS a063836e372f4021b8f5f7d65ff18c6b (127.0.98.2:39859): tablet e9c2795042b7483199b73736de2c143b (table test-table [id=fc64f26ccca54f9097fd75c38073a701]) successfully deleted
W20260620 14:19:32.647944 7821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:32.647799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.0.1:35504)
0620 14:19:32.647862 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:32.647890 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35681: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:19:32.648078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 7459
W20260620 14:19:32.648077 7072 catalog_manager.cc:4584] TS acf0c092f8ac435da3112368a2180822 (127.0.98.1:35681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9c2795042b7483199b73736de2c143b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35681: connect: Connection reset by peer (error 104)
I20260620 14:19:32.652331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 7595
I20260620 14:19:32.659188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 6953
I20260620 14:19:32.664017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 7141
2026-06-20T14:19:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6926 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:32 dist-test-slave-h6n5 krb5kdc[7851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:32 dist-test-slave-h6n5 krb5kdc[7851](info): set up 2 sockets
Jun 20 14:19:32 dist-test-slave-h6n5 krb5kdc[7851](info): commencing operation
krb5kdc: starting...
W20260620 14:19:33.700188 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:33 dist-test-slave-h6n5 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:19:33.818079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46671
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:33.897945 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:33.898154 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:33.898175 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:33.899293 7864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:33.899331 7864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:19:33.899344 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:33.899358 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:33.899369 7864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:33.899381 7864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:33.900792 7864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46671
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:33.900988 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:33.901155 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:33.906069 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:33.906077 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:33.906077 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:33.906368 7864 server_base.cc:1034] running on GCE node
I20260620 14:19:33.906633 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:33.906935 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:33.908110 7864 hybrid_clock.cc:648] HybridClock initialized: now 1781965173908093 us; error 28 us; skew 500 ppm
Jun 20 14:19:33 dist-test-slave-h6n5 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965173, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:33.915489 7864 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:19:33.916127 7864 webserver.cc:460] Webserver started at http://127.0.98.62:32809/ using document root <none> and password file <none>
I20260620 14:19:33.916471 7864 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:33.916535 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:33.916688 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:33.917948 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "a6ea0cee0de046db97e65f4bfcc692f1"
format_stamp: "Formatted at 2026-06-20 14:19:33 on dist-test-slave-h6n5"
server_key: "cdf7e3a5dffc8c1997362b8e17289a00"
server_key_iv: "b87147af22c1a7c12f12341a52f7f4dd"
server_key_version: "encryptionkey@0"
I20260620 14:19:33.918371 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6ea0cee0de046db97e65f4bfcc692f1"
format_stamp: "Formatted at 2026-06-20 14:19:33 on dist-test-slave-h6n5"
server_key: "cdf7e3a5dffc8c1997362b8e17289a00"
server_key_iv: "b87147af22c1a7c12f12341a52f7f4dd"
server_key_version: "encryptionkey@0"
I20260620 14:19:33.920171 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 14:19:33.921067 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:33.921284 7864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:33.921368 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "a6ea0cee0de046db97e65f4bfcc692f1"
format_stamp: "Formatted at 2026-06-20 14:19:33 on dist-test-slave-h6n5"
server_key: "cdf7e3a5dffc8c1997362b8e17289a00"
server_key_iv: "b87147af22c1a7c12f12341a52f7f4dd"
server_key_version: "encryptionkey@0"
I20260620 14:19:33.921438 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:33.935300 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:33.937611 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:33.937723 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:33.942276 7864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46671
I20260620 14:19:33.942318 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46671 every 8 connection(s)
I20260620 14:19:33.942685 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:33.943300 7942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:33.945655 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Bootstrap starting.
I20260620 14:19:33.946241 7942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:33.946564 7942 log.cc:824] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:33.947244 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: No bootstrap required, opened a new log
I20260620 14:19:33.949358 7942 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260620 14:19:33.949501 7942 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:33.949523 7942 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ea0cee0de046db97e65f4bfcc692f1, State: Initialized, Role: FOLLOWER
I20260620 14:19:33.949704 7942 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260620 14:19:33.949769 7942 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:33.949812 7942 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:33.949961 7942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:33.950613 7942 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260620 14:19:33.950704 7942 leader_election.cc:302] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ea0cee0de046db97e65f4bfcc692f1; no voters:
I20260620 14:19:33.950892 7942 leader_election.cc:288] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:33.951085 7945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:33.951141 7942 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:33.951244 7945 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [term 1 LEADER]: Becoming Leader. State: Replica: a6ea0cee0de046db97e65f4bfcc692f1, State: Running, Role: LEADER
I20260620 14:19:33.951332 7945 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260620 14:19:33.951822 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ea0cee0de046db97e65f4bfcc692f1. Latest consensus state: current_term: 1 leader_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } } }
I20260620 14:19:33.951843 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea0cee0de046db97e65f4bfcc692f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } } }
I20260620 14:19:33.951931 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:33.951961 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:33.952124 7954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:33.952759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 7864
I20260620 14:19:33.952848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:33.953047 392 external_mini_cluster.cc:1428] Setting key e7ddc98ff5d6a633bd1c01a43d02b02a
I20260620 14:19:33.953343 7954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:33.955706 7954 catalog_manager.cc:1347] Generated new cluster ID: c51ddcde35994b56a91a7647688901b9
I20260620 14:19:33.955760 7954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:33 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:33.963533 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:33.953732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.0.98.62:46671)
0620 14:19:33.953911 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:33.953920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:33.954217 (+ 297us) server_negotiation.cc:407] Connection header received
0620 14:19:33.954358 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:33.954365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:33.954421 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:33.954582 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:33.955331 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:33.956420 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:33.957227 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:33.957650 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:33.957779 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:33.960232 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:33.960236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:33.960237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:33.960248 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:33.961944 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:33.962329 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:33.962333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:33.962333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:33.962425 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:33.962637 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:33.962638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:33.962638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:33.962871 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:33.963049 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:33.963118 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:19:33.963370 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260620 14:19:33.967895 7954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:33.968444 7954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:33.974311 7954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Generated new TSK 0
I20260620 14:19:33.974522 7954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:19:34.010818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:34.107491 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:34.107796 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:34.107837 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:34.109068 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:34.109139 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:34.109176 7968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:34.109213 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:34.110749 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:34.111037 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:34.111276 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:34.116545 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:34.116576 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:34.116734 7968 server_base.cc:1034] running on GCE node
W20260620 14:19:34.116835 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:34.117058 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:34.117318 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:34.118484 7968 hybrid_clock.cc:648] HybridClock initialized: now 1781965174118471 us; error 49 us; skew 500 ppm
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:34.121121 7968 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:19:34.121598 7968 webserver.cc:460] Webserver started at http://127.0.98.1:44865/ using document root <none> and password file <none>
I20260620 14:19:34.121836 7968 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:34.121902 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:34.122056 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:34.123128 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1e1fc63684a4e809a72e426a3f47340"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "ef38c95504dae9b73d8bb3aa661089ab"
server_key_iv: "ad00b43a087d865731a018c0dc70e7fd"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.123497 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e1fc63684a4e809a72e426a3f47340"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "ef38c95504dae9b73d8bb3aa661089ab"
server_key_iv: "ad00b43a087d865731a018c0dc70e7fd"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.125063 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:34.125815 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.126055 7968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:34.126166 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "c1e1fc63684a4e809a72e426a3f47340"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "ef38c95504dae9b73d8bb3aa661089ab"
server_key_iv: "ad00b43a087d865731a018c0dc70e7fd"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.126238 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:34.137609 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:34.139477 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:34.139714 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:34.140326 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:34.140389 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.140429 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:34.140458 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:34.147462 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38111
I20260620 14:19:34.147567 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38111 every 8 connection(s)
I20260620 14:19:34.147915 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:34.150374 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.140719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55847 (local address 127.0.98.62:46671)
0620 14:19:34.140784 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.140786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.141014 (+ 228us) server_negotiation.cc:407] Connection header received
0620 14:19:34.141524 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.141526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.141542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.141622 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.142345 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.143192 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.144347 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.144574 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.144663 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.147268 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.147269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.147270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.147278 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.148977 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.149388 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.149389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.149390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.149450 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.149702 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.149703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.149703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.149824 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.149917 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.150187 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.150280 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:19:34.150552 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.140724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.1:55847)
0620 14:19:34.140868 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.140902 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.141397 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.141674 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.141677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.141920 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:34.142242 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.142249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.143289 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.143290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.144250 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.144252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.144685 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.144686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.144802 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.145350 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:34.145360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:34.147165 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:34.149062 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.149069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.149071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.149291 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.149516 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.149516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.149517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.149632 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.149911 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:34.149917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:34.150056 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.150166 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.150322 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260620 14:19:34.151013 8099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46671
I20260620 14:19:34.151146 8099 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:34.151335 8099 heartbeater.cc:511] Master 127.0.98.62:46671 requested a full tablet report, sending...
I20260620 14:19:34.152240 7895 ts_manager.cc:194] Registered new tserver with Master: c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111)
I20260620 14:19:34.153115 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55847
I20260620 14:19:34.156617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 7968
I20260620 14:19:34.156741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:34.156939 392 external_mini_cluster.cc:1428] Setting key c512e37f2ef0c39d17a199804c3aa381
I20260620 14:19:34.157783 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.152075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.1:39265)
0620 14:19:34.152258 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.152270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.152347 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.152501 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.152502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.152690 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:34.152750 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.152752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.153497 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.153498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.154048 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.154050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.154372 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.154373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.154424 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.154945 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:34.154949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:34.155215 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:34.156762 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.156763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.156763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.156965 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.157172 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.157173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.157173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.157230 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.157457 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:34.157458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:34.157504 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.157573 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.157674 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260620 14:19:34.157799 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.152161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39265 (local address 127.0.98.62:46671)
0620 14:19:34.152219 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.152221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.152323 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:19:34.152423 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.152424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.152434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.152486 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.152851 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.153408 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.154125 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.154304 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.154361 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.155312 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.155313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.155313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.155319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.156657 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.157045 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.157046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.157047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.157104 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.157288 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.157288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.157289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.157393 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.157448 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.157624 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.157709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:19:34.182664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:34.264621 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:34.264880 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:34.264919 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:34.266132 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:34.266211 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:34.266245 8108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:34.266283 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:34.267867 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:34.268159 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:34.268397 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:34.273764 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:34.273823 8108 server_base.cc:1034] running on GCE node
W20260620 14:19:34.273766 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:34.273830 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:34.274169 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:34.274421 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:34.275612 8108 hybrid_clock.cc:648] HybridClock initialized: now 1781965174275587 us; error 40 us; skew 500 ppm
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:34.278225 8108 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:19:34.278687 8108 webserver.cc:460] Webserver started at http://127.0.98.2:39211/ using document root <none> and password file <none>
I20260620 14:19:34.278936 8108 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:34.279006 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:34.279155 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:34.280277 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "63982894098c4bb4bae2bbe9cc2b210c"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "7a5ae48b989b9be61a8ef65c23d5520d"
server_key_iv: "906e65bf5efec3507edf20faf2ea87d4"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.280649 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "63982894098c4bb4bae2bbe9cc2b210c"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "7a5ae48b989b9be61a8ef65c23d5520d"
server_key_iv: "906e65bf5efec3507edf20faf2ea87d4"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.282027 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:34.282829 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.283011 8108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:34.283111 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "63982894098c4bb4bae2bbe9cc2b210c"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "7a5ae48b989b9be61a8ef65c23d5520d"
server_key_iv: "906e65bf5efec3507edf20faf2ea87d4"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.283192 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:34.295740 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:34.298105 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:34.298274 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:34.298799 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:34.298882 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.298949 8108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:34.298980 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.305678 8108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42161
I20260620 14:19:34.305738 8238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42161 every 8 connection(s)
I20260620 14:19:34.306092 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:34.307794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 8108
I20260620 14:19:34.307910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:34.308125 392 external_mini_cluster.cc:1428] Setting key 5070cea1b2b1b1cc30a4dc7609ff7827
I20260620 14:19:34.309873 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.299392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49643 (local address 127.0.98.62:46671)
0620 14:19:34.299449 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.299452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.299705 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:19:34.300260 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.300261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.300274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.300338 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.301227 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.302087 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.303705 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.304017 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.304133 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.306794 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.306795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.306796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.306803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.308382 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.308802 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.308803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.308804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.308867 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.309195 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.309196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.309196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.309316 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.309382 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.309744 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.309813 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:19:34.310148 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.299373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.2:49643)
0620 14:19:34.299533 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.299608 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.300148 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.300411 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.300415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.300784 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:34.301138 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.301145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.302180 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.302181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.303607 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.303609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.304146 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.304147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.304304 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.304919 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:34.304928 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:34.306702 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:34.308441 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.308448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.308450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.308730 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.308943 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.308943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.308944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.309099 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.309394 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:34.309400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:34.309583 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.309698 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.309877 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260620 14:19:34.310665 8239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46671
I20260620 14:19:34.310766 8239 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:34.310961 8239 heartbeater.cc:511] Master 127.0.98.62:46671 requested a full tablet report, sending...
I20260620 14:19:34.311314 7896 ts_manager.cc:194] Registered new tserver with Master: 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161)
I20260620 14:19:34.311847 7896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49643
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:19:34.317811 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.311941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44417 (local address 127.0.98.62:46671)
0620 14:19:34.311984 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.311987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.312141 (+ 154us) server_negotiation.cc:407] Connection header received
0620 14:19:34.312187 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.312188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.312197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.312251 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.312576 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.313158 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.313861 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.314068 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.314152 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.315081 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.315082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.315083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.315089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.316575 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.316930 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.316931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.316932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.316986 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.317208 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.317209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.317209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.317358 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.317435 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.317641 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.317719 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:19:34.317826 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.311882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.2:44417)
0620 14:19:34.312064 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.312074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.312133 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.312257 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.312258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.312442 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:34.312493 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.312494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.313253 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.313254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.313765 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.313767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.314155 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.314156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.314225 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.314685 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:34.314691 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:34.314970 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:34.316669 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.316670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.316670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.316843 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.317076 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.317077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.317078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.317137 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.317441 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:34.317442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:34.317488 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.317611 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.317721 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:19:34.334949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:34.415980 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:34.416182 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:34.416205 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:34.417332 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:34.417371 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:34.417392 8248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:19:34.417448 8248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:34.418881 8248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:34.419116 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:34.419348 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:34.424300 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:34.424263 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:34.424336 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:34.424620 8248 server_base.cc:1034] running on GCE node
I20260620 14:19:34.424818 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:34.425007 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:34.426178 8248 hybrid_clock.cc:648] HybridClock initialized: now 1781965174426173 us; error 36 us; skew 500 ppm
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:34.428681 8248 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:19:34.429185 8248 webserver.cc:460] Webserver started at http://127.0.98.3:37181/ using document root <none> and password file <none>
I20260620 14:19:34.429466 8248 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:34.429512 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:34.429668 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:34.430740 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "0bf339bf39a84d97a862fe2ac0be0780"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "36aa9b96a62a8bc0306e42c09abbc87a"
server_key_iv: "52dd7aaeb877b96c58494ee640cd5847"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.431115 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0bf339bf39a84d97a862fe2ac0be0780"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "36aa9b96a62a8bc0306e42c09abbc87a"
server_key_iv: "52dd7aaeb877b96c58494ee640cd5847"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.432474 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:34.433202 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.433367 8248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:34.433449 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "0bf339bf39a84d97a862fe2ac0be0780"
format_stamp: "Formatted at 2026-06-20 14:19:34 on dist-test-slave-h6n5"
server_key: "36aa9b96a62a8bc0306e42c09abbc87a"
server_key_iv: "52dd7aaeb877b96c58494ee640cd5847"
server_key_version: "encryptionkey@0"
I20260620 14:19:34.433495 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:34.440111 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:34.441996 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:34.442101 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:34.442495 8248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:34.442528 8248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.442554 8248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:34.442608 8248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.449152 8248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33199
I20260620 14:19:34.449198 8378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33199 every 8 connection(s)
I20260620 14:19:34.449512 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:34.453190 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.442991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45397 (local address 127.0.98.62:46671)
0620 14:19:34.443061 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.443074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.443242 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:19:34.443803 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.443805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.443819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.443896 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.444717 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.445559 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.447120 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.447470 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.447518 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.450342 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.450345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.450345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.450353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.451982 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.452403 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.452404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.452404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.452454 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.452709 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.452709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.452710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.452811 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.452864 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.453071 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.453131 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:34.453365 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.442968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.3:45397)
0620 14:19:34.443116 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.443154 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.443707 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.443982 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.443985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.444330 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:34.444611 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.444617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.445663 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.445664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.447010 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.447013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.447646 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.447646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.447749 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.448352 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:34.448362 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:34.450191 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:34.452051 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.452056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.452057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.452336 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.452528 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.452529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.452529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.452630 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.452856 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:34.452859 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:34.452978 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.453056 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.453171 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260620 14:19:34.453876 8379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46671
I20260620 14:19:34.453958 8379 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:34.454140 8379 heartbeater.cc:511] Master 127.0.98.62:46671 requested a full tablet report, sending...
I20260620 14:19:34.454540 7895 ts_manager.cc:194] Registered new tserver with Master: 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199)
I20260620 14:19:34.454953 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45397
I20260620 14:19:34.459756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 8248
I20260620 14:19:34.459887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:34.460142 392 external_mini_cluster.cc:1428] Setting key 1c80b1bc8c00a1ea1a4468eab091e250
I20260620 14:19:34.460176 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.454985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55899 (local address 127.0.98.62:46671)
0620 14:19:34.455030 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.455031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.455087 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:19:34.455157 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.455157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.455167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.455223 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.455536 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.456018 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.456709 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.456906 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.456979 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.457781 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.457783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.457783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.457790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.459005 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.459359 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.459361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.459361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.459418 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.459662 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.459663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.459663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.459773 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.459849 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.460014 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.460085 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:19:34.460191 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.454899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.3:55899)
0620 14:19:34.455027 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.455038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.455100 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.455228 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.455228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.455407 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:34.455460 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.455462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.456114 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.456115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.456620 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.456621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.456980 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.456981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.457045 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.457435 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:34.457439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:34.457696 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:34.459099 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.459100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.459100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.459273 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.459522 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.459522 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.459523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.459610 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.459840 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:34.459841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:34.459883 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.459987 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.460086 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260620 14:19:34.460918 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:19:34.482003 7964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:34.476750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.98.62:46671)
0620 14:19:34.476817 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.476819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.476830 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:34.476870 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.476871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.476879 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.476931 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.477492 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.477988 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.478701 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.478885 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.478933 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.481899 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35250: BlockingRecv error: recv got EOF from 127.0.0.1:35250 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:34.490074 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.482937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.0.98.62:46671)
0620 14:19:34.483045 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.483048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.483152 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:19:34.483230 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.483232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.483242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.483315 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.483881 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.484346 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.485043 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.485224 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.485299 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.487661 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.487662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.487663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.487669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.488911 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.489279 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.489281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.489281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.489336 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.489607 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.489608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.489608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.489729 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.489819 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.489922 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.490010 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:19:34.491863 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:34.493454 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:34.499804 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.495837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.98.2:42161)
0620 14:19:34.496003 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.496008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.496203 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:19:34.496253 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.496255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.496267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.496359 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:34.496529 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.497638 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.498733 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.499283 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.499356 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.499442 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.499666 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.499708 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260620 14:19:34.499809 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.495734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42161 (local address 127.0.0.1:57050)
0620 14:19:34.496016 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.496059 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.496123 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.496374 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.496375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.496377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:34.496433 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.496435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.497734 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.497735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.498659 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.498662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.499408 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.499409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.499498 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.499515 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.499646 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.499703 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":58,"threads_started":1}
I20260620 14:19:34.500013 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.495845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.0.98.3:33199)
0620 14:19:34.496139 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.496144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.496182 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:19:34.496221 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.496223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.496240 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.496318 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:34.496507 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.497407 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.498575 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.499446 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.499518 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.499614 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.499810 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.499858 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":74,"threads_started":1}
I20260620 14:19:34.500265 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.495736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33199 (local address 127.0.0.1:45540)
0620 14:19:34.496056 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.496065 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.496123 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.496334 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.496337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.496339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:34.496404 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.496406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.497486 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.497486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.498476 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.498478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.499647 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.499648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.499716 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.499720 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.500140 (+ 420us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.500192 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260620 14:19:34.500268 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.495950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.0.1:59272)
0620 14:19:34.496452 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.496463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.496501 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.496835 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.496836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.496838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:34.496872 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.496873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.498036 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.498037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.499097 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.499100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.500061 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.500062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.500124 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.500130 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.500183 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.500212 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":63,"threads_started":1}
I20260620 14:19:34.500473 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.496100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.0.98.1:38111)
0620 14:19:34.496677 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.496682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.496691 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:34.496730 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.496732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.496745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.496822 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:34.496931 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.497952 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.499190 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.499991 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.500059 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.500123 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.500310 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.500370 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":76,"threads_started":1}
I20260620 14:19:34.500895 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 2640ee9623894b4fb3cebf0c108cb963 (DEFAULT_TABLE table=test-table [id=c6321f8fdbfd4072ab9019bd206bbec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:34.501006 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 2640ee9623894b4fb3cebf0c108cb963 (DEFAULT_TABLE table=test-table [id=c6321f8fdbfd4072ab9019bd206bbec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:34.501276 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2640ee9623894b4fb3cebf0c108cb963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.501276 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2640ee9623894b4fb3cebf0c108cb963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.504094 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 2640ee9623894b4fb3cebf0c108cb963 (DEFAULT_TABLE table=test-table [id=c6321f8fdbfd4072ab9019bd206bbec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:34.504431 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2640ee9623894b4fb3cebf0c108cb963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.505118 8407 tablet_bootstrap.cc:492] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Bootstrap starting.
I20260620 14:19:34.505435 8408 tablet_bootstrap.cc:492] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Bootstrap starting.
I20260620 14:19:34.505965 8407 tablet_bootstrap.cc:654] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.505973 8408 tablet_bootstrap.cc:654] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.506281 8408 log.cc:824] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:34.506300 8407 log.cc:824] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:34.507009 8408 tablet_bootstrap.cc:492] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: No bootstrap required, opened a new log
I20260620 14:19:34.507094 8408 ts_tablet_manager.cc:1397] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:34.507098 8407 tablet_bootstrap.cc:492] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: No bootstrap required, opened a new log
I20260620 14:19:34.507167 8407 ts_tablet_manager.cc:1397] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:34.508339 8411 tablet_bootstrap.cc:492] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Bootstrap starting.
I20260620 14:19:34.509016 8411 tablet_bootstrap.cc:654] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.509012 8407 raft_consensus.cc:348] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.509021 8408 raft_consensus.cc:348] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.509192 8408 raft_consensus.cc:374] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.509192 8407 raft_consensus.cc:374] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.509236 8407 raft_consensus.cc:729] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf339bf39a84d97a862fe2ac0be0780, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.509236 8408 raft_consensus.cc:729] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.509342 8411 log.cc:824] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:34.509346 8407 consensus_queue.cc:260] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.509383 8408 consensus_queue.cc:260] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.509639 8408 ts_tablet_manager.cc:1428] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:19:34.509644 8407 ts_tablet_manager.cc:1428] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:34.509649 8379 heartbeater.cc:503] Master 127.0.98.62:46671 was elected leader, sending a full tablet report...
I20260620 14:19:34.509809 8239 heartbeater.cc:503] Master 127.0.98.62:46671 was elected leader, sending a full tablet report...
I20260620 14:19:34.510586 8411 tablet_bootstrap.cc:492] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: No bootstrap required, opened a new log
I20260620 14:19:34.510704 8411 ts_tablet_manager.cc:1397] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:34.512284 8411 raft_consensus.cc:348] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.512432 8411 raft_consensus.cc:374] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.512477 8411 raft_consensus.cc:729] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e1fc63684a4e809a72e426a3f47340, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.512591 8411 consensus_queue.cc:260] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.512838 8411 ts_tablet_manager.cc:1428] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:34.512885 8099 heartbeater.cc:503] Master 127.0.98.62:46671 was elected leader, sending a full tablet report...
W20260620 14:19:34.556639 8240 tablet.cc:2369] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:34.644536 8412 raft_consensus.cc:482] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:34.644657 8412 raft_consensus.cc:504] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.645072 8412 leader_election.cc:288] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161), c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111)
I20260620 14:19:34.648073 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.645375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44557 (local address 127.0.98.1:38111)
0620 14:19:34.645425 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.645427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.645603 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:19:34.645683 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.645684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.645704 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.645770 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:34.645940 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.646390 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.647299 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.647686 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.647768 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.647779 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.647954 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.647991 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:19:34.648080 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.645465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56897 (local address 127.0.98.2:42161)
0620 14:19:34.645534 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.645536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.645543 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:34.645566 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.645566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.645575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.645649 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:34.645826 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.646314 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.647276 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.647704 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.647785 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.647795 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.647971 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.648007 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:19:34.648114 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.645319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42161 (local address 127.0.98.3:56897)
0620 14:19:34.645365 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.645377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.645438 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.645679 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.645681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.645683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:34.645752 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.645753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.646420 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.646421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.647177 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.647179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.647783 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.647784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.647883 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.647885 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.647965 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.648007 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":2}
I20260620 14:19:34.648133 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.645319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.98.3:44557)
0620 14:19:34.645527 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.645543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.645616 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.645785 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.645786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.645787 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:34.645866 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.645867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.646477 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.646478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.647202 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.647204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.647780 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.647781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.647876 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.647879 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.647950 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.647990 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":65,"thread_start_us":108,"threads_started":1}
I20260620 14:19:34.648474 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2640ee9623894b4fb3cebf0c108cb963" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340" is_pre_election: true
I20260620 14:19:34.648630 8053 raft_consensus.cc:2454] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 0.
I20260620 14:19:34.648804 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2640ee9623894b4fb3cebf0c108cb963" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63982894098c4bb4bae2bbe9cc2b210c" is_pre_election: true
I20260620 14:19:34.648850 8266 leader_election.cc:302] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf339bf39a84d97a862fe2ac0be0780, c1e1fc63684a4e809a72e426a3f47340; no voters:
I20260620 14:19:34.648962 8193 raft_consensus.cc:2454] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 0.
W20260620 14:19:34.648980 8100 tablet.cc:2369] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:34.649048 8412 raft_consensus.cc:2781] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:34.649106 8412 raft_consensus.cc:482] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:34.649137 8412 raft_consensus.cc:3037] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.650132 8412 raft_consensus.cc:504] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.650287 8412 leader_election.cc:288] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 election: Requested vote from peers 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161), c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111)
I20260620 14:19:34.650432 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2640ee9623894b4fb3cebf0c108cb963" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63982894098c4bb4bae2bbe9cc2b210c"
I20260620 14:19:34.650524 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2640ee9623894b4fb3cebf0c108cb963" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340"
I20260620 14:19:34.650554 8193 raft_consensus.cc:3037] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.650621 8053 raft_consensus.cc:3037] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.651484 8193 raft_consensus.cc:2454] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 1.
I20260620 14:19:34.651484 8053 raft_consensus.cc:2454] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 1.
I20260620 14:19:34.651701 8264 leader_election.cc:302] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf339bf39a84d97a862fe2ac0be0780, 63982894098c4bb4bae2bbe9cc2b210c; no voters:
I20260620 14:19:34.651808 8412 raft_consensus.cc:2781] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:34.651949 8412 raft_consensus.cc:686] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf339bf39a84d97a862fe2ac0be0780, State: Running, Role: LEADER
I20260620 14:19:34.652050 8412 consensus_queue.cc:237] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } }
I20260620 14:19:34.653030 7896 catalog_manager.cc:5495] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0bf339bf39a84d97a862fe2ac0be0780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260620 14:19:34.683820 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.681190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.0.98.3:33199)
0620 14:19:34.681265 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.681267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.681278 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:34.681322 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.681323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.681352 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.681427 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:34.681584 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.682119 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.682890 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.683155 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.683239 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.683440 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:34.683605 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:34.683688 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.683713 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.683733 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:19:34.685637 8053 raft_consensus.cc:1261] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Refusing update from remote peer 0bf339bf39a84d97a862fe2ac0be0780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:34.685660 8193 raft_consensus.cc:1261] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Refusing update from remote peer 0bf339bf39a84d97a862fe2ac0be0780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:34.686025 8412 consensus_queue.cc:1035] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:34.686184 8417 consensus_queue.cc:1035] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:34.688573 8420 mvcc.cc:205] Tried to move back new op lower bound from 7298929355509751808 to 7298929355375529984. Current Snapshot: MvccSnapshot[applied={T|T < 7298929355509751808}]
I20260620 14:19:34.688817 8421 mvcc.cc:205] Tried to move back new op lower bound from 7298929355509751808 to 7298929355375529984. Current Snapshot: MvccSnapshot[applied={T|T < 7298929355509751808}]
I20260620 14:19:34.688884 8422 mvcc.cc:205] Tried to move back new op lower bound from 7298929355509751808 to 7298929355375529984. Current Snapshot: MvccSnapshot[applied={T|T < 7298929355509751808}]
I20260620 14:19:34.691412 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:34.691514 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:34.692296 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Sending DeleteTablet for 3 replicas of tablet 2640ee9623894b4fb3cebf0c108cb963
I20260620 14:19:34.692589 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 2640ee9623894b4fb3cebf0c108cb963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45540
I20260620 14:19:34.692670 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 2640ee9623894b4fb3cebf0c108cb963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59272
I20260620 14:19:34.692751 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 2640ee9623894b4fb3cebf0c108cb963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57050
I20260620 14:19:34.692894 8431 tablet_replica.cc:330] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: stopping tablet replica
I20260620 14:19:34.692960 8430 tablet_replica.cc:330] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: stopping tablet replica
I20260620 14:19:34.693006 8431 raft_consensus.cc:2229] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:34.693048 8430 raft_consensus.cc:2229] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:34.693122 8431 raft_consensus.cc:2258] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:34.693197 8430 raft_consensus.cc:2258] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:34.693341 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:34.693485 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:34.693588 8431 ts_tablet_manager.cc:1905] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:34.693643 8430 ts_tablet_manager.cc:1905] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:34.694612 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 94e070adfac24406893f490e31effcdd (DEFAULT_TABLE table=test-table [id=7afdeb71163d47aab536a36521a9f1b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:34.694720 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e070adfac24406893f490e31effcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.695384 8431 ts_tablet_manager.cc:1918] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:34.695425 8430 ts_tablet_manager.cc:1918] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:34.695452 8431 log.cc:1196] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/2640ee9623894b4fb3cebf0c108cb963
I20260620 14:19:34.695494 8430 log.cc:1196] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/2640ee9623894b4fb3cebf0c108cb963
I20260620 14:19:34.695741 8431 ts_tablet_manager.cc:1939] T 2640ee9623894b4fb3cebf0c108cb963 P c1e1fc63684a4e809a72e426a3f47340: Deleting consensus metadata
I20260620 14:19:34.695777 8430 ts_tablet_manager.cc:1939] T 2640ee9623894b4fb3cebf0c108cb963 P 0bf339bf39a84d97a862fe2ac0be0780: Deleting consensus metadata
I20260620 14:19:34.696166 8408 tablet_bootstrap.cc:492] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: Bootstrap starting.
I20260620 14:19:34.696625 8408 tablet_bootstrap.cc:654] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.696635 7882 catalog_manager.cc:4841] TS c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111): tablet 2640ee9623894b4fb3cebf0c108cb963 (table test-table [id=c6321f8fdbfd4072ab9019bd206bbec0]) successfully deleted
I20260620 14:19:34.696707 7882 catalog_manager.cc:4841] TS 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199): tablet 2640ee9623894b4fb3cebf0c108cb963 (table test-table [id=c6321f8fdbfd4072ab9019bd206bbec0]) successfully deleted
I20260620 14:19:34.696750 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 94e070adfac24406893f490e31effcdd (DEFAULT_TABLE table=test-table [id=7afdeb71163d47aab536a36521a9f1b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:34.696852 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e070adfac24406893f490e31effcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.697049 8432 tablet_replica.cc:330] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: stopping tablet replica
I20260620 14:19:34.697180 8432 raft_consensus.cc:2229] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:34.697341 8432 raft_consensus.cc:2258] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:34.697729 8432 ts_tablet_manager.cc:1905] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:34.698062 8407 tablet_bootstrap.cc:492] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Bootstrap starting.
I20260620 14:19:34.698161 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 94e070adfac24406893f490e31effcdd (DEFAULT_TABLE table=test-table [id=7afdeb71163d47aab536a36521a9f1b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:34.698266 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e070adfac24406893f490e31effcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.698513 8407 tablet_bootstrap.cc:654] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.699297 8432 ts_tablet_manager.cc:1918] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:34.699375 8432 log.cc:1196] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/2640ee9623894b4fb3cebf0c108cb963
I20260620 14:19:34.699508 8407 tablet_bootstrap.cc:492] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: No bootstrap required, opened a new log
I20260620 14:19:34.699515 8411 tablet_bootstrap.cc:492] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: Bootstrap starting.
I20260620 14:19:34.699579 8407 ts_tablet_manager.cc:1397] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:34.699681 8432 ts_tablet_manager.cc:1939] T 2640ee9623894b4fb3cebf0c108cb963 P 63982894098c4bb4bae2bbe9cc2b210c: Deleting consensus metadata
I20260620 14:19:34.699738 8407 raft_consensus.cc:348] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.699810 8407 raft_consensus.cc:374] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.699829 8407 raft_consensus.cc:729] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf339bf39a84d97a862fe2ac0be0780, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.699872 8407 consensus_queue.cc:260] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.699951 8407 ts_tablet_manager.cc:1428] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.699962 8411 tablet_bootstrap.cc:654] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.700070 7880 catalog_manager.cc:4841] TS 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161): tablet 2640ee9623894b4fb3cebf0c108cb963 (table test-table [id=c6321f8fdbfd4072ab9019bd206bbec0]) successfully deleted
I20260620 14:19:34.700176 8408 tablet_bootstrap.cc:492] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: No bootstrap required, opened a new log
I20260620 14:19:34.700259 8408 ts_tablet_manager.cc:1397] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
W20260620 14:19:34.700286 8380 tablet.cc:2369] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:34.700404 8408 raft_consensus.cc:348] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.700464 8408 raft_consensus.cc:374] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.700488 8408 raft_consensus.cc:729] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.700543 8408 consensus_queue.cc:260] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.700578 8411 tablet_bootstrap.cc:492] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: No bootstrap required, opened a new log
I20260620 14:19:34.700632 8408 ts_tablet_manager.cc:1428] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.700636 8411 ts_tablet_manager.cc:1397] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:34.700778 8411 raft_consensus.cc:348] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.700850 8411 raft_consensus.cc:374] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.700875 8411 raft_consensus.cc:729] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e1fc63684a4e809a72e426a3f47340, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.700917 8411 consensus_queue.cc:260] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.701210 8411 ts_tablet_manager.cc:1428] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:19:34.711829 8413 raft_consensus.cc:482] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:34.711933 8413 raft_consensus.cc:504] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.712265 8413 leader_election.cc:288] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199)
I20260620 14:19:34.715462 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.712652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60289 (local address 127.0.98.3:33199)
0620 14:19:34.712696 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.712698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.712742 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:19:34.712809 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.712810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.712819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.712869 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:34.713059 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.713590 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.714742 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.715157 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.715226 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.715236 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.715361 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.715393 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:19:34.715487 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.712582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33199 (local address 127.0.98.2:60289)
0620 14:19:34.712690 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.712701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.712757 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.712910 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.712911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.712912 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:34.712976 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.712977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.713689 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.713689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.714650 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.714653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.715222 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.715223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.715292 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.715297 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.715348 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.715389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260620 14:19:34.715811 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e070adfac24406893f490e31effcdd" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf339bf39a84d97a862fe2ac0be0780" is_pre_election: true
I20260620 14:19:34.715945 8333 raft_consensus.cc:2454] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 0.
I20260620 14:19:34.716034 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.712456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.98.2:59099)
0620 14:19:34.712659 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.712671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.712725 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.712869 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.712870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.712871 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:34.712945 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.712947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.713888 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.713889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.715036 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.715039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.715794 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.715795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.715874 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.715878 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.715931 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.715975 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260620 14:19:34.716198 8126 leader_election.cc:302] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf339bf39a84d97a862fe2ac0be0780, 63982894098c4bb4bae2bbe9cc2b210c; no voters:
I20260620 14:19:34.716295 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.712522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59099 (local address 127.0.98.1:38111)
0620 14:19:34.712587 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.712590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.712780 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:19:34.712807 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.712808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.712817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.712866 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:34.713018 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.713787 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.715131 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.715721 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.715793 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.715803 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.716171 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.716226 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:19:34.716321 8413 raft_consensus.cc:2781] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:34.716348 8413 raft_consensus.cc:482] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:34.716368 8413 raft_consensus.cc:3037] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.716467 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e070adfac24406893f490e31effcdd" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340" is_pre_election: true
I20260620 14:19:34.716547 8053 raft_consensus.cc:2454] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 0.
I20260620 14:19:34.717249 8413 raft_consensus.cc:504] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.717382 8413 leader_election.cc:288] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 election: Requested vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199)
I20260620 14:19:34.717571 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e070adfac24406893f490e31effcdd" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340"
I20260620 14:19:34.717586 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e070adfac24406893f490e31effcdd" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf339bf39a84d97a862fe2ac0be0780"
I20260620 14:19:34.717664 8333 raft_consensus.cc:3037] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.717666 8053 raft_consensus.cc:3037] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.718281 8053 raft_consensus.cc:2454] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 1.
I20260620 14:19:34.718441 8126 leader_election.cc:302] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63982894098c4bb4bae2bbe9cc2b210c, c1e1fc63684a4e809a72e426a3f47340; no voters:
I20260620 14:19:34.718503 8333 raft_consensus.cc:2454] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 1.
I20260620 14:19:34.718528 8413 raft_consensus.cc:2781] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:34.718655 8413 raft_consensus.cc:686] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 1 LEADER]: Becoming Leader. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Running, Role: LEADER
I20260620 14:19:34.718734 8413 consensus_queue.cc:237] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.719455 7895 catalog_manager.cc:5495] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:19:34 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965173, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:34.752679 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.744399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.0.1:35276)
0620 14:19:34.744592 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.744605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.744684 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.744839 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.744840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.745250 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:34.745316 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.745319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.746002 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.746003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.746874 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.746877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.747229 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.747230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.747293 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.747960 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:34.747965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:34.749644 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:34.751385 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.751393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.751396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.751700 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.751919 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:34.751920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:34.751920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:34.751992 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:34.752300 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:34.752307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:34.752361 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.752434 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.752565 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260620 14:19:34.752787 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.744508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.0.98.62:46671)
0620 14:19:34.744560 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.744562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.744680 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:19:34.744748 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.744749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.744757 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.744809 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:34.745419 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.745921 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.746968 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.747151 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.747206 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.749738 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:34.749739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:34.749739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:34.749746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:34.751305 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.751773 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.751774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.751774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.751840 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:34.752059 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:34.752060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:34.752060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:34.752210 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:34.752270 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.752479 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.752657 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":29}
I20260620 14:19:34.755174 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35276:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:19:34.755376 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:34.756819 8033 tablet_service.cc:1467] Processing CreateTablet for tablet dead1c861b5e4b39b35a70d864da4f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=135b615e389f418e9c094e2fe5189f10]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:34.756846 8173 tablet_service.cc:1467] Processing CreateTablet for tablet dead1c861b5e4b39b35a70d864da4f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=135b615e389f418e9c094e2fe5189f10]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:34.756882 8313 tablet_service.cc:1467] Processing CreateTablet for tablet dead1c861b5e4b39b35a70d864da4f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=135b615e389f418e9c094e2fe5189f10]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:19:34.756973 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dead1c861b5e4b39b35a70d864da4f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.756973 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dead1c861b5e4b39b35a70d864da4f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.757015 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dead1c861b5e4b39b35a70d864da4f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:34.758451 8408 tablet_bootstrap.cc:492] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: Bootstrap starting.
I20260620 14:19:34.758848 8408 tablet_bootstrap.cc:654] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.759186 8411 tablet_bootstrap.cc:492] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: Bootstrap starting.
I20260620 14:19:34.759594 8411 tablet_bootstrap.cc:654] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.759657 8407 tablet_bootstrap.cc:492] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: Bootstrap starting.
I20260620 14:19:34.760041 8408 tablet_bootstrap.cc:492] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: No bootstrap required, opened a new log
I20260620 14:19:34.760051 8407 tablet_bootstrap.cc:654] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:34.760088 8408 ts_tablet_manager.cc:1397] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:19:34.760505 8408 raft_consensus.cc:348] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.760613 8408 raft_consensus.cc:374] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.760639 8408 raft_consensus.cc:729] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.760659 8407 tablet_bootstrap.cc:492] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: No bootstrap required, opened a new log
I20260620 14:19:34.760699 8407 ts_tablet_manager.cc:1397] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:34.760682 8408 consensus_queue.cc:260] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.760782 8408 ts_tablet_manager.cc:1428] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:34.760939 8413 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.760977 8407 raft_consensus.cc:348] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.761019 8413 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:34.761052 8407 raft_consensus.cc:374] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.761088 8407 raft_consensus.cc:729] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf339bf39a84d97a862fe2ac0be0780, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.761130 8407 consensus_queue.cc:260] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.761236 8407 ts_tablet_manager.cc:1428] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:34.761281 8425 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.761376 8425 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:34.761446 8411 tablet_bootstrap.cc:492] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: No bootstrap required, opened a new log
I20260620 14:19:34.761497 8411 ts_tablet_manager.cc:1397] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:34.761643 8411 raft_consensus.cc:348] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.761693 8411 raft_consensus.cc:374] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:34.761705 8411 raft_consensus.cc:729] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e1fc63684a4e809a72e426a3f47340, State: Initialized, Role: FOLLOWER
I20260620 14:19:34.761762 8411 consensus_queue.cc:260] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.761863 8411 ts_tablet_manager.cc:1428] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:34.761920 8415 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.762002 8415 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:34.966696 8413 raft_consensus.cc:482] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:34.966795 8413 raft_consensus.cc:504] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.966990 8413 leader_election.cc:288] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199)
I20260620 14:19:34.967253 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead1c861b5e4b39b35a70d864da4f1d" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340" is_pre_election: true
I20260620 14:19:34.967360 8053 raft_consensus.cc:2454] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 0.
I20260620 14:19:34.967314 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead1c861b5e4b39b35a70d864da4f1d" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf339bf39a84d97a862fe2ac0be0780" is_pre_election: true
I20260620 14:19:34.967469 8333 raft_consensus.cc:2454] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 0.
I20260620 14:19:34.967581 8126 leader_election.cc:302] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63982894098c4bb4bae2bbe9cc2b210c, c1e1fc63684a4e809a72e426a3f47340; no voters:
I20260620 14:19:34.967742 8413 raft_consensus.cc:2781] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:34.967803 8413 raft_consensus.cc:482] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:34.967836 8413 raft_consensus.cc:3037] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.968536 8413 raft_consensus.cc:504] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.968641 8413 leader_election.cc:288] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 election: Requested vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199)
I20260620 14:19:34.968825 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead1c861b5e4b39b35a70d864da4f1d" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340"
I20260620 14:19:34.968876 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead1c861b5e4b39b35a70d864da4f1d" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf339bf39a84d97a862fe2ac0be0780"
I20260620 14:19:34.968910 8053 raft_consensus.cc:3037] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.968942 8333 raft_consensus.cc:3037] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:34.969683 8333 raft_consensus.cc:2454] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 1.
I20260620 14:19:34.969677 8053 raft_consensus.cc:2454] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 1.
I20260620 14:19:34.969901 8126 leader_election.cc:302] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf339bf39a84d97a862fe2ac0be0780, 63982894098c4bb4bae2bbe9cc2b210c; no voters:
I20260620 14:19:34.970011 8413 raft_consensus.cc:2781] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:34.970065 8413 raft_consensus.cc:686] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [term 1 LEADER]: Becoming Leader. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Running, Role: LEADER
I20260620 14:19:34.970118 8413 consensus_queue.cc:237] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:34.970389 8437 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: TxnStatusTablet state changed. Reason: New leader 63982894098c4bb4bae2bbe9cc2b210c. Latest consensus state: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.970485 8437 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:34.970645 8449 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:19:34.970793 8449 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:19:34.970721 7895 catalog_manager.cc:5495] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } health_report { overall_health: UNKNOWN } } }
I20260620 14:19:34.991843 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.988817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42161 (local address 127.0.0.1:57054)
0620 14:19:34.988891 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:34.988904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:34.988959 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:34.989276 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:34.989277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:34.989278 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:34.989334 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.989336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.990274 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.990274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.990839 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:34.990842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.991317 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.991318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:34.991373 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.991386 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:34.991696 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:34.991704 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:19:34.991744 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 14:19:34.991783 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:19:34.991853 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.989054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.0.98.2:42161)
0620 14:19:34.989122 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.989124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.989134 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:34.989168 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.989169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.989189 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.989264 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:34.989411 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.990185 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.990922 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.991232 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.991302 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:34.991478 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:34.991628 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:34.991702 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:34.991750 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:19:34.991771 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:19:34.993085 8333 raft_consensus.cc:1261] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Refusing update from remote peer 63982894098c4bb4bae2bbe9cc2b210c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:34.993163 8053 raft_consensus.cc:1261] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Refusing update from remote peer 63982894098c4bb4bae2bbe9cc2b210c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:34.993494 8437 consensus_queue.cc:1035] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:34.993645 8413 consensus_queue.cc:1035] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:34.994208 8415 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: TxnStatusTablet state changed. Reason: New leader 63982894098c4bb4bae2bbe9cc2b210c. Latest consensus state: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.994225 8425 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: TxnStatusTablet state changed. Reason: New leader 63982894098c4bb4bae2bbe9cc2b210c. Latest consensus state: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.994297 8415 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:34.994310 8425 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:34.995251 8437 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.995347 8437 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:34.995795 8413 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.995874 8413 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P 63982894098c4bb4bae2bbe9cc2b210c: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:19:34.996047 8425 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.996127 8425 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P 0bf339bf39a84d97a862fe2ac0be0780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:34.996184 8415 tablet_replica.cc:439] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } } }
I20260620 14:19:34.996259 8415 tablet_replica.cc:442] T dead1c861b5e4b39b35a70d864da4f1d P c1e1fc63684a4e809a72e426a3f47340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:19:35.000752 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:34.998366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.0.98.2:42161)
0620 14:19:34.998412 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:34.998415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:34.998425 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:34.998459 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:34.998460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:34.998470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:34.998515 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:34.998611 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:34.999093 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:34.999936 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.000236 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.000321 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:35.000507 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:35.000540 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:35.000591 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:35.000656 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:19:35.000674 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:19:35.006429 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:35.003507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45447 (local address 127.0.98.2:42161)
0620 14:19:35.003572 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:35.003574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:35.003731 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:19:35.003756 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:35.003757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:35.003766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:35.003822 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:35.004118 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.004575 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.005710 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.005886 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.005951 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:35.006139 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:35.006165 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:35.006204 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:35.006325 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:19:35.006346 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":21}
I20260620 14:19:35.006559 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:35.003400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42161 (local address 127.0.98.2:45447)
0620 14:19:35.003621 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:35.003632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:35.003698 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:35.003842 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:35.003843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:35.003844 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:19:35.003906 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:35.003908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.004655 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.004656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:35.005607 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:35.005610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.005965 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.005966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:35.006033 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:35.006052 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:19:35.006247 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:19:35.006249 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:35.006325 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:19:35.006374 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260620 14:19:35.008909 8333 raft_consensus.cc:1261] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Refusing update from remote peer 63982894098c4bb4bae2bbe9cc2b210c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:35.008929 8053 raft_consensus.cc:1261] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Refusing update from remote peer 63982894098c4bb4bae2bbe9cc2b210c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:35.009161 8457 consensus_queue.cc:1035] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:35.009271 8437 consensus_queue.cc:1035] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:35.020534 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:35.020664 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:35.021351 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Sending DeleteTablet for 3 replicas of tablet 94e070adfac24406893f490e31effcdd
I20260620 14:19:35.021590 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e070adfac24406893f490e31effcdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57050
I20260620 14:19:35.021595 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e070adfac24406893f490e31effcdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59272
I20260620 14:19:35.021685 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e070adfac24406893f490e31effcdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45540
I20260620 14:19:35.021710 8431 tablet_replica.cc:330] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: stopping tablet replica
I20260620 14:19:35.021714 8432 tablet_replica.cc:330] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: stopping tablet replica
I20260620 14:19:35.021762 8431 raft_consensus.cc:2229] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:35.021770 8432 raft_consensus.cc:2229] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:35.021775 8430 tablet_replica.cc:330] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: stopping tablet replica
I20260620 14:19:35.021804 8431 raft_consensus.cc:2258] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.021833 8430 raft_consensus.cc:2229] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:35.021858 8432 raft_consensus.cc:2258] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.021867 8430 raft_consensus.cc:2258] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.022164 8432 ts_tablet_manager.cc:1905] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.022164 8430 ts_tablet_manager.cc:1905] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.022248 8431 ts_tablet_manager.cc:1905] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.024107 8431 ts_tablet_manager.cc:1918] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:35.024166 8431 log.cc:1196] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/94e070adfac24406893f490e31effcdd
I20260620 14:19:35.024379 8431 ts_tablet_manager.cc:1939] T 94e070adfac24406893f490e31effcdd P c1e1fc63684a4e809a72e426a3f47340: Deleting consensus metadata
I20260620 14:19:35.024386 8430 ts_tablet_manager.cc:1918] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:35.024432 8430 log.cc:1196] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/94e070adfac24406893f490e31effcdd
I20260620 14:19:35.024652 8430 ts_tablet_manager.cc:1939] T 94e070adfac24406893f490e31effcdd P 0bf339bf39a84d97a862fe2ac0be0780: Deleting consensus metadata
I20260620 14:19:35.024703 8432 ts_tablet_manager.cc:1918] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:35.024755 8432 log.cc:1196] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/94e070adfac24406893f490e31effcdd
I20260620 14:19:35.024961 8432 ts_tablet_manager.cc:1939] T 94e070adfac24406893f490e31effcdd P 63982894098c4bb4bae2bbe9cc2b210c: Deleting consensus metadata
I20260620 14:19:35.025422 7882 catalog_manager.cc:4841] TS 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199): tablet 94e070adfac24406893f490e31effcdd (table test-table [id=7afdeb71163d47aab536a36521a9f1b3]) successfully deleted
I20260620 14:19:35.025538 7882 catalog_manager.cc:4841] TS c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111): tablet 94e070adfac24406893f490e31effcdd (table test-table [id=7afdeb71163d47aab536a36521a9f1b3]) successfully deleted
I20260620 14:19:35.025681 7880 catalog_manager.cc:4841] TS 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161): tablet 94e070adfac24406893f490e31effcdd (table test-table [id=7afdeb71163d47aab536a36521a9f1b3]) successfully deleted
Jun 20 14:19:35 dist-test-slave-h6n5 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:19:35 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:35 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:19:35.046489 7964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:35.040309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.98.62:46671)
0620 14:19:35.040384 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:35.040387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:35.040542 (+ 155us) server_negotiation.cc:407] Connection header received
0620 14:19:35.040622 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:35.040623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:35.040632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:35.040681 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:35.041380 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.041861 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.042920 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.043104 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.043156 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:35.046406 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35278: BlockingRecv error: recv got EOF from 127.0.0.1:35278 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 14:19:35 dist-test-slave-h6n5 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:19:35.054518 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:35.047240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.0.98.62:46671)
0620 14:19:35.047280 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:35.047283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:35.047309 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:19:35.047382 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:35.047383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:35.047393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:35.047453 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:35.047925 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.048641 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.049285 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.049569 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.049645 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:35.052018 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:35.052019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:35.052020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:35.052026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:35.053399 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:35.053805 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:35.053806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:35.053806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:35.053854 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:35.054044 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:35.054044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:35.054044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:35.054182 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:35.054260 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:35.054402 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:19:35.054455 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:35.056059 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:35.056234 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:35.057586 8173 tablet_service.cc:1467] Processing CreateTablet for tablet fd4c2e73d7d846b29fad43017558e258 (DEFAULT_TABLE table=test-table [id=0f8d2a351d2047a8a7900f48396ad91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:35.057654 8033 tablet_service.cc:1467] Processing CreateTablet for tablet fd4c2e73d7d846b29fad43017558e258 (DEFAULT_TABLE table=test-table [id=0f8d2a351d2047a8a7900f48396ad91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:35.057654 8313 tablet_service.cc:1467] Processing CreateTablet for tablet fd4c2e73d7d846b29fad43017558e258 (DEFAULT_TABLE table=test-table [id=0f8d2a351d2047a8a7900f48396ad91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:35.057722 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4c2e73d7d846b29fad43017558e258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:35.057770 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4c2e73d7d846b29fad43017558e258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:35.057770 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4c2e73d7d846b29fad43017558e258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:35.059039 8407 tablet_bootstrap.cc:492] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: Bootstrap starting.
I20260620 14:19:35.059039 8411 tablet_bootstrap.cc:492] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: Bootstrap starting.
I20260620 14:19:35.059139 8408 tablet_bootstrap.cc:492] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: Bootstrap starting.
I20260620 14:19:35.059401 8407 tablet_bootstrap.cc:654] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:35.059420 8411 tablet_bootstrap.cc:654] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:35.059644 8408 tablet_bootstrap.cc:654] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:35.060148 8411 tablet_bootstrap.cc:492] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: No bootstrap required, opened a new log
I20260620 14:19:35.060197 8411 ts_tablet_manager.cc:1397] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:35.060236 8407 tablet_bootstrap.cc:492] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: No bootstrap required, opened a new log
I20260620 14:19:35.060281 8407 ts_tablet_manager.cc:1397] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:35.060348 8411 raft_consensus.cc:348] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.060395 8408 tablet_bootstrap.cc:492] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: No bootstrap required, opened a new log
I20260620 14:19:35.060411 8411 raft_consensus.cc:374] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:35.060380 8407 raft_consensus.cc:348] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.060434 8411 raft_consensus.cc:729] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e1fc63684a4e809a72e426a3f47340, State: Initialized, Role: FOLLOWER
I20260620 14:19:35.060438 8407 raft_consensus.cc:374] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:35.060436 8408 ts_tablet_manager.cc:1397] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:19:35.060461 8407 raft_consensus.cc:729] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf339bf39a84d97a862fe2ac0be0780, State: Initialized, Role: FOLLOWER
I20260620 14:19:35.060479 8411 consensus_queue.cc:260] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.060498 8407 consensus_queue.cc:260] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.060577 8407 ts_tablet_manager.cc:1428] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:35.060567 8411 ts_tablet_manager.cc:1428] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:35.060587 8408 raft_consensus.cc:348] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.060652 8408 raft_consensus.cc:374] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:35.060672 8408 raft_consensus.cc:729] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Initialized, Role: FOLLOWER
I20260620 14:19:35.060714 8408 consensus_queue.cc:260] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.060796 8408 ts_tablet_manager.cc:1428] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:35.123629 8425 raft_consensus.cc:482] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:35.123754 8425 raft_consensus.cc:504] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.123936 8425 leader_election.cc:288] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161)
I20260620 14:19:35.124188 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c2e73d7d846b29fad43017558e258" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340" is_pre_election: true
I20260620 14:19:35.124177 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c2e73d7d846b29fad43017558e258" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63982894098c4bb4bae2bbe9cc2b210c" is_pre_election: true
I20260620 14:19:35.124310 8053 raft_consensus.cc:2454] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 0.
I20260620 14:19:35.124306 8193 raft_consensus.cc:2454] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 0.
I20260620 14:19:35.124467 8266 leader_election.cc:302] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf339bf39a84d97a862fe2ac0be0780, c1e1fc63684a4e809a72e426a3f47340; no voters:
I20260620 14:19:35.124588 8425 raft_consensus.cc:2781] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:35.124621 8425 raft_consensus.cc:482] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:35.124650 8425 raft_consensus.cc:3037] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:35.125479 8425 raft_consensus.cc:504] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.125617 8425 leader_election.cc:288] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 election: Requested vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161)
I20260620 14:19:35.125778 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c2e73d7d846b29fad43017558e258" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63982894098c4bb4bae2bbe9cc2b210c"
I20260620 14:19:35.125855 8193 raft_consensus.cc:3037] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:35.125874 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4c2e73d7d846b29fad43017558e258" candidate_uuid: "0bf339bf39a84d97a862fe2ac0be0780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340"
I20260620 14:19:35.125958 8053 raft_consensus.cc:3037] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:35.126521 8053 raft_consensus.cc:2454] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 1.
I20260620 14:19:35.126627 8193 raft_consensus.cc:2454] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf339bf39a84d97a862fe2ac0be0780 in term 1.
I20260620 14:19:35.126683 8266 leader_election.cc:302] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf339bf39a84d97a862fe2ac0be0780, c1e1fc63684a4e809a72e426a3f47340; no voters:
I20260620 14:19:35.126783 8425 raft_consensus.cc:2781] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:35.126844 8425 raft_consensus.cc:686] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf339bf39a84d97a862fe2ac0be0780, State: Running, Role: LEADER
I20260620 14:19:35.126897 8425 consensus_queue.cc:237] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.127458 7894 catalog_manager.cc:5495] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0bf339bf39a84d97a862fe2ac0be0780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } health_report { overall_health: HEALTHY } } }
I20260620 14:19:35.138727 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:35.136121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.0.98.3:33199)
0620 14:19:35.136171 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:35.136173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:35.136183 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:35.136244 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:35.136245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:35.136259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:35.136331 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:35.136433 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.137155 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.137951 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.138251 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.138322 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:35.138490 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:35.138524 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:35.138577 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:35.138630 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:19:35.138651 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:19:35.139539 8193 raft_consensus.cc:1261] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Refusing update from remote peer 0bf339bf39a84d97a862fe2ac0be0780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:35.139626 8053 raft_consensus.cc:1261] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Refusing update from remote peer 0bf339bf39a84d97a862fe2ac0be0780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:35.139762 8425 consensus_queue.cc:1035] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:35.139851 8412 consensus_queue.cc:1035] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:35.142670 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:35.142757 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:35.143363 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Sending DeleteTablet for 3 replicas of tablet fd4c2e73d7d846b29fad43017558e258
I20260620 14:19:35.143676 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4c2e73d7d846b29fad43017558e258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59272
I20260620 14:19:35.143703 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4c2e73d7d846b29fad43017558e258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45540
I20260620 14:19:35.143708 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4c2e73d7d846b29fad43017558e258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57050
I20260620 14:19:35.143772 8431 tablet_replica.cc:330] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: stopping tablet replica
I20260620 14:19:35.143787 8430 tablet_replica.cc:330] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: stopping tablet replica
I20260620 14:19:35.143791 8432 tablet_replica.cc:330] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: stopping tablet replica
I20260620 14:19:35.143823 8431 raft_consensus.cc:2229] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:35.143837 8430 raft_consensus.cc:2229] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:35.143838 8432 raft_consensus.cc:2229] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:35.143863 8431 raft_consensus.cc:2258] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.143877 8432 raft_consensus.cc:2258] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.143939 8430 raft_consensus.cc:2258] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.144184 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:35.144323 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:35.144345 8430 ts_tablet_manager.cc:1905] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.144685 8432 ts_tablet_manager.cc:1905] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.145771 8431 ts_tablet_manager.cc:1905] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.146065 8432 ts_tablet_manager.cc:1918] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:35.146126 8432 log.cc:1196] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/fd4c2e73d7d846b29fad43017558e258
I20260620 14:19:35.146356 8432 ts_tablet_manager.cc:1939] T fd4c2e73d7d846b29fad43017558e258 P 63982894098c4bb4bae2bbe9cc2b210c: Deleting consensus metadata
I20260620 14:19:35.146697 7880 catalog_manager.cc:4841] TS 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161): tablet fd4c2e73d7d846b29fad43017558e258 (table test-table [id=0f8d2a351d2047a8a7900f48396ad91c]) successfully deleted
I20260620 14:19:35.147501 8431 ts_tablet_manager.cc:1918] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:35.147593 8431 log.cc:1196] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/fd4c2e73d7d846b29fad43017558e258
I20260620 14:19:35.147792 8431 ts_tablet_manager.cc:1939] T fd4c2e73d7d846b29fad43017558e258 P c1e1fc63684a4e809a72e426a3f47340: Deleting consensus metadata
I20260620 14:19:35.148120 7882 catalog_manager.cc:4841] TS c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111): tablet fd4c2e73d7d846b29fad43017558e258 (table test-table [id=0f8d2a351d2047a8a7900f48396ad91c]) successfully deleted
I20260620 14:19:35.148422 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 72d4f5e866324197923e8b123057421e (DEFAULT_TABLE table=test-table [id=2943975380ee406e8aad8374989626d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:35.148581 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 72d4f5e866324197923e8b123057421e (DEFAULT_TABLE table=test-table [id=2943975380ee406e8aad8374989626d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:35.148664 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d4f5e866324197923e8b123057421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:35.148820 8430 ts_tablet_manager.cc:1918] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:35.148871 8430 log.cc:1196] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/fd4c2e73d7d846b29fad43017558e258
I20260620 14:19:35.149080 8430 ts_tablet_manager.cc:1939] T fd4c2e73d7d846b29fad43017558e258 P 0bf339bf39a84d97a862fe2ac0be0780: Deleting consensus metadata
I20260620 14:19:35.149721 8411 tablet_bootstrap.cc:492] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340: Bootstrap starting.
I20260620 14:19:35.150110 8411 tablet_bootstrap.cc:654] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:35.150348 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 72d4f5e866324197923e8b123057421e (DEFAULT_TABLE table=test-table [id=2943975380ee406e8aad8374989626d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:35.150445 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d4f5e866324197923e8b123057421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:35.151582 8408 tablet_bootstrap.cc:492] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: Bootstrap starting.
I20260620 14:19:35.151880 8408 tablet_bootstrap.cc:654] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:35.152109 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d4f5e866324197923e8b123057421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:35.153254 8407 tablet_bootstrap.cc:492] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: Bootstrap starting.
I20260620 14:19:35.153563 8408 tablet_bootstrap.cc:492] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: No bootstrap required, opened a new log
I20260620 14:19:35.153587 8407 tablet_bootstrap.cc:654] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:35.153613 8408 ts_tablet_manager.cc:1397] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 14:19:35.153720 8408 raft_consensus.cc:348] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.153791 8408 raft_consensus.cc:374] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:35.153810 8408 raft_consensus.cc:729] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Initialized, Role: FOLLOWER
I20260620 14:19:35.153849 8408 consensus_queue.cc:260] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.153962 8408 ts_tablet_manager.cc:1428] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:35.154165 8407 tablet_bootstrap.cc:492] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: No bootstrap required, opened a new log
I20260620 14:19:35.154229 8407 ts_tablet_manager.cc:1397] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:35.154326 8411 tablet_bootstrap.cc:492] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340: No bootstrap required, opened a new log
I20260620 14:19:35.154337 8407 raft_consensus.cc:348] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.154376 8411 ts_tablet_manager.cc:1397] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 14:19:35.154412 8407 raft_consensus.cc:374] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:35.154467 8407 raft_consensus.cc:729] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf339bf39a84d97a862fe2ac0be0780, State: Initialized, Role: FOLLOWER
I20260620 14:19:35.154513 8407 consensus_queue.cc:260] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.154510 8411 raft_consensus.cc:348] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.154563 8411 raft_consensus.cc:374] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:35.154588 8411 raft_consensus.cc:729] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e1fc63684a4e809a72e426a3f47340, State: Initialized, Role: FOLLOWER
I20260620 14:19:35.154604 8407 ts_tablet_manager.cc:1428] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:35.154623 8411 consensus_queue.cc:260] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.154700 8411 ts_tablet_manager.cc:1428] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:35.154965 7882 catalog_manager.cc:4841] TS 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199): tablet fd4c2e73d7d846b29fad43017558e258 (table test-table [id=0f8d2a351d2047a8a7900f48396ad91c]) successfully deleted
I20260620 14:19:35.347659 8437 raft_consensus.cc:482] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:35.347780 8437 raft_consensus.cc:504] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.347971 8437 leader_election.cc:288] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199)
I20260620 14:19:35.348186 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4f5e866324197923e8b123057421e" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340" is_pre_election: true
I20260620 14:19:35.348202 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4f5e866324197923e8b123057421e" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf339bf39a84d97a862fe2ac0be0780" is_pre_election: true
I20260620 14:19:35.348289 8333 raft_consensus.cc:2454] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 0.
I20260620 14:19:35.348289 8053 raft_consensus.cc:2454] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 0.
I20260620 14:19:35.348438 8126 leader_election.cc:302] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63982894098c4bb4bae2bbe9cc2b210c, c1e1fc63684a4e809a72e426a3f47340; no voters:
I20260620 14:19:35.348587 8437 raft_consensus.cc:2781] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:35.348637 8437 raft_consensus.cc:482] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:35.348675 8437 raft_consensus.cc:3037] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:35.349422 8437 raft_consensus.cc:504] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.349551 8437 leader_election.cc:288] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 election: Requested vote from peers c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111), 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199)
I20260620 14:19:35.349786 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4f5e866324197923e8b123057421e" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e1fc63684a4e809a72e426a3f47340"
I20260620 14:19:35.349786 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4f5e866324197923e8b123057421e" candidate_uuid: "63982894098c4bb4bae2bbe9cc2b210c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf339bf39a84d97a862fe2ac0be0780"
I20260620 14:19:35.349877 8053 raft_consensus.cc:3037] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:35.349877 8333 raft_consensus.cc:3037] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:35.350641 8053 raft_consensus.cc:2454] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 1.
I20260620 14:19:35.350634 8333 raft_consensus.cc:2454] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63982894098c4bb4bae2bbe9cc2b210c in term 1.
I20260620 14:19:35.350816 8126 leader_election.cc:302] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf339bf39a84d97a862fe2ac0be0780, 63982894098c4bb4bae2bbe9cc2b210c; no voters:
I20260620 14:19:35.350947 8437 raft_consensus.cc:2781] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:35.351011 8437 raft_consensus.cc:686] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 1 LEADER]: Becoming Leader. State: Replica: 63982894098c4bb4bae2bbe9cc2b210c, State: Running, Role: LEADER
I20260620 14:19:35.351079 8437 consensus_queue.cc:237] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } }
I20260620 14:19:35.351715 7895 catalog_manager.cc:5495] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "63982894098c4bb4bae2bbe9cc2b210c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63982894098c4bb4bae2bbe9cc2b210c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 } health_report { overall_health: UNKNOWN } } }
I20260620 14:19:35.385425 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:35.383576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.0.98.2:42161)
0620 14:19:35.383614 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:35.383616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:35.383639 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:19:35.383705 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:35.383706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:35.383713 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:35.383762 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:35.383847 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.384293 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.384942 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:35.385113 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:35.385162 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:35.385269 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:35.385292 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:35.385326 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:35.385358 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:19:35.385371 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:19:35.387603 8333 raft_consensus.cc:1261] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Refusing update from remote peer 63982894098c4bb4bae2bbe9cc2b210c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:35.387604 8053 raft_consensus.cc:1261] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Refusing update from remote peer 63982894098c4bb4bae2bbe9cc2b210c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:35.387804 8437 consensus_queue.cc:1035] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e1fc63684a4e809a72e426a3f47340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:35.387903 8413 consensus_queue.cc:1035] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf339bf39a84d97a862fe2ac0be0780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:35.397689 7893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:35.397790 7893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:35.398401 7893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ea0cee0de046db97e65f4bfcc692f1: Sending DeleteTablet for 3 replicas of tablet 72d4f5e866324197923e8b123057421e
I20260620 14:19:35.398622 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d4f5e866324197923e8b123057421e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57050
I20260620 14:19:35.398649 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d4f5e866324197923e8b123057421e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45540
I20260620 14:19:35.398646 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d4f5e866324197923e8b123057421e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59272
I20260620 14:19:35.398733 8430 tablet_replica.cc:330] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: stopping tablet replica
I20260620 14:19:35.398733 8431 tablet_replica.cc:330] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340: stopping tablet replica
I20260620 14:19:35.398777 8431 raft_consensus.cc:2229] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:35.398777 8430 raft_consensus.cc:2229] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:35.398808 8431 raft_consensus.cc:2258] T 72d4f5e866324197923e8b123057421e P c1e1fc63684a4e809a72e426a3f47340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.398808 8430 raft_consensus.cc:2258] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.398845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 7968
I20260620 14:19:35.400143 8430 ts_tablet_manager.cc:1905] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.400187 8432 tablet_replica.cc:330] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: stopping tablet replica
I20260620 14:19:35.400240 8432 raft_consensus.cc:2229] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:35.400295 8432 raft_consensus.cc:2258] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:35.400576 8432 ts_tablet_manager.cc:1905] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:35.401906 8430 ts_tablet_manager.cc:1918] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:35.401966 8430 log.cc:1196] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/72d4f5e866324197923e8b123057421e
I20260620 14:19:35.402165 8432 ts_tablet_manager.cc:1918] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:19:35.402202 8430 ts_tablet_manager.cc:1939] T 72d4f5e866324197923e8b123057421e P 0bf339bf39a84d97a862fe2ac0be0780: Deleting consensus metadata
I20260620 14:19:35.402211 8432 log.cc:1196] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/72d4f5e866324197923e8b123057421e
I20260620 14:19:35.402406 8432 ts_tablet_manager.cc:1939] T 72d4f5e866324197923e8b123057421e P 63982894098c4bb4bae2bbe9cc2b210c: Deleting consensus metadata
I20260620 14:19:35.402570 7882 catalog_manager.cc:4841] TS 0bf339bf39a84d97a862fe2ac0be0780 (127.0.98.3:33199): tablet 72d4f5e866324197923e8b123057421e (table test-table [id=2943975380ee406e8aad8374989626d7]) successfully deleted
I20260620 14:19:35.402751 7880 catalog_manager.cc:4841] TS 63982894098c4bb4bae2bbe9cc2b210c (127.0.98.2:42161): tablet 72d4f5e866324197923e8b123057421e (table test-table [id=2943975380ee406e8aad8374989626d7]) successfully deleted
W20260620 14:19:35.404325 7882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38111 (error 108)
I20260620 14:19:35.404580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 8108
I20260620 14:19:35.405675 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:35.404648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38111 (local address 127.0.0.1:40190)
0620 14:19:35.405582 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:35.405627 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":37,"threads_started":1}
W20260620 14:19:35.405813 7882 catalog_manager.cc:4584] TS c1e1fc63684a4e809a72e426a3f47340 (127.0.98.1:38111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72d4f5e866324197923e8b123057421e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38111: connect: Connection refused (error 111)
I20260620 14:19:35.411026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 8248
I20260620 14:19:35.416599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 7864
2026-06-20T14:19:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2751 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:35 dist-test-slave-h6n5 krb5kdc[8491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:35 dist-test-slave-h6n5 krb5kdc[8491](info): set up 2 sockets
Jun 20 14:19:35 dist-test-slave-h6n5 krb5kdc[8491](info): commencing operation
krb5kdc: starting...
W20260620 14:19:36.460214 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:36 dist-test-slave-h6n5 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:36.578565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:36.659998 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:36.660245 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:36.660284 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:36.661473 8504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:36.661548 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:36.661576 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:36.661608 8504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:36.661638 8504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:36.663182 8504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:36.663460 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:36.663722 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:36.668717 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:36.668731 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:36.668740 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:36.668951 8504 server_base.cc:1034] running on GCE node
I20260620 14:19:36.669247 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:36.669551 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:36.670717 8504 hybrid_clock.cc:648] HybridClock initialized: now 1781965176670704 us; error 47 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:19:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1258 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:36 dist-test-slave-h6n5 krb5kdc[8517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:36 dist-test-slave-h6n5 krb5kdc[8517](info): set up 2 sockets
Jun 20 14:19:36 dist-test-slave-h6n5 krb5kdc[8517](info): commencing operation
krb5kdc: starting...
W20260620 14:19:37.724144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:37 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:37.841975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:37.925925 8530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:37.926138 8530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:37.926162 8530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:37.927371 8530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:37.927418 8530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:37.927433 8530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:37.927444 8530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:37.927457 8530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:37.929029 8530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:37.929281 8530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:37.929508 8530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:37.934840 8536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:37.934832 8535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:37.934852 8538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:37.935159 8530 server_base.cc:1034] running on GCE node
I20260620 14:19:37.935407 8530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:37.935753 8530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:37.936914 8530 hybrid_clock.cc:648] HybridClock initialized: now 1781965177936894 us; error 38 us; skew 500 ppm
Jun 20 14:19:37 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:37.939375 8530 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:37.939898 8530 webserver.cc:460] Webserver started at http://127.0.98.62:36349/ using document root <none> and password file <none>
I20260620 14:19:37.940155 8530 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:37.940223 8530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:37.940367 8530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:37.941447 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "570f4d960a4b421c8241992f6c8f0a08"
format_stamp: "Formatted at 2026-06-20 14:19:37 on dist-test-slave-h6n5"
server_key: "ac516e90131edd288e394787b2fbcf35"
server_key_iv: "6f49bd90df9600267b92cf83b18c9ad8"
server_key_version: "encryptionkey@0"
I20260620 14:19:37.941824 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "570f4d960a4b421c8241992f6c8f0a08"
format_stamp: "Formatted at 2026-06-20 14:19:37 on dist-test-slave-h6n5"
server_key: "ac516e90131edd288e394787b2fbcf35"
server_key_iv: "6f49bd90df9600267b92cf83b18c9ad8"
server_key_version: "encryptionkey@0"
I20260620 14:19:37.943222 8530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:37.944043 8545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:37.944269 8530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:19:37.944379 8530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "570f4d960a4b421c8241992f6c8f0a08"
format_stamp: "Formatted at 2026-06-20 14:19:37 on dist-test-slave-h6n5"
server_key: "ac516e90131edd288e394787b2fbcf35"
server_key_iv: "6f49bd90df9600267b92cf83b18c9ad8"
server_key_version: "encryptionkey@0"
I20260620 14:19:37.944456 8530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:37.981441 8530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:37.983685 8530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:37.983876 8530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:37.987906 8530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45111
I20260620 14:19:37.987939 8597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45111 every 8 connection(s)
I20260620 14:19:37.988443 8530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:37.989130 8598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:37.991356 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Bootstrap starting.
I20260620 14:19:37.992004 8598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:37.992252 8598 log.cc:824] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:37.992980 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: No bootstrap required, opened a new log
I20260620 14:19:37.994544 8598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:37.994699 8598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:37.994722 8598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570f4d960a4b421c8241992f6c8f0a08, State: Initialized, Role: FOLLOWER
I20260620 14:19:37.994828 8598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:37.994957 8598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:37.995007 8598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:37.995060 8598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:37.996142 8598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:37.996268 8598 leader_election.cc:302] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 570f4d960a4b421c8241992f6c8f0a08; no voters:
I20260620 14:19:37.996492 8598 leader_election.cc:288] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:37.996601 8601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:37.996762 8601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 LEADER]: Becoming Leader. State: Replica: 570f4d960a4b421c8241992f6c8f0a08, State: Running, Role: LEADER
I20260620 14:19:37.996766 8598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:37.996940 8601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:37.997396 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "570f4d960a4b421c8241992f6c8f0a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } } }
I20260620 14:19:37.997521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 8530
I20260620 14:19:37.997653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:37.997682 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:37.997866 392 external_mini_cluster.cc:1428] Setting key 867b44ba3934f702a4136dad98d1e51f
I20260620 14:19:37.997599 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 570f4d960a4b421c8241992f6c8f0a08. Latest consensus state: current_term: 1 leader_uuid: "570f4d960a4b421c8241992f6c8f0a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } } }
I20260620 14:19:37.997982 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:37.998044 8610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:37.998746 8610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:38.000361 8610 catalog_manager.cc:1347] Generated new cluster ID: 84220fc7c3ec491599131fc7a7e29a3e
I20260620 14:19:38.000412 8610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:38.009991 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:37.998341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.98.62:45111)
0620 14:19:37.998809 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:37.998819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:37.998855 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:19:37.999026 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:37.999033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:37.999103 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:37.999248 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:38.000049 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.001468 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.002542 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.003206 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.003328 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.006280 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:38.006284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:38.006284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:38.006297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:38.008323 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.008726 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.008729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.008729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.008838 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.009059 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.009060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.009060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.009350 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:38.009483 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:38.009618 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:19:38.009745 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":166,"threads_started":1}
I20260620 14:19:38.016219 8610 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:38.016731 8610 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:38.023820 8610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Generated new TSK 0
I20260620 14:19:38.024003 8610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:38.050182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45111
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:38.132099 8624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:38.132298 8624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:38.132321 8624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:38.133580 8624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:38.133631 8624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:38.133658 8624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:38.135241 8624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:38.135484 8624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:38.135759 8624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:38.140789 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.140795 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:38.140830 8624 server_base.cc:1034] running on GCE node
W20260620 14:19:38.140795 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:38.141183 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:38.141408 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:38.142634 8624 hybrid_clock.cc:648] HybridClock initialized: now 1781965178142609 us; error 36 us; skew 500 ppm
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:38.145154 8624 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:38.145586 8624 webserver.cc:460] Webserver started at http://127.0.98.1:42675/ using document root <none> and password file <none>
I20260620 14:19:38.145813 8624 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:38.145880 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:38.146044 8624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:38.147145 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "cabe73b42a5e4aca83eeef3b476a69ae"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "3abf8c895025db6dbc357f88b2b5ec8f"
server_key_iv: "2a4d37fe8a9005a7d316c6ccaacf8546"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.147528 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cabe73b42a5e4aca83eeef3b476a69ae"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "3abf8c895025db6dbc357f88b2b5ec8f"
server_key_iv: "2a4d37fe8a9005a7d316c6ccaacf8546"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.148952 8624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:38.149770 8639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.149964 8624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:38.150091 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "cabe73b42a5e4aca83eeef3b476a69ae"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "3abf8c895025db6dbc357f88b2b5ec8f"
server_key_iv: "2a4d37fe8a9005a7d316c6ccaacf8546"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.150180 8624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:38.174386 8624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:38.176604 8624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:38.176772 8624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:38.177026 8624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:38.177340 8624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:38.177393 8624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.177430 8624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:38.177480 8624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.183039 8624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38567
I20260620 14:19:38.183139 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38567 every 8 connection(s)
I20260620 14:19:38.183439 8624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:38.185370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 8624
I20260620 14:19:38.185498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:38.185683 392 external_mini_cluster.cc:1428] Setting key 1095a6a37a0ff147961f55a2989fc6a5
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:19:38.193892 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.184281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45897 (local address 127.0.98.62:45111)
0620 14:19:38.184343 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:38.184345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:38.184555 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:19:38.185716 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:38.185734 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:38.185747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:38.185822 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:38.186585 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.187093 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.188418 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.188641 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.188731 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.190988 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:38.190989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:38.190990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:38.190997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:38.192519 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.192927 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.192928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.192928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.192990 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.193240 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.193241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.193242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.193357 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:38.193432 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:38.193711 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:19:38.193797 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:19:38.194085 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.184255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:45897)
0620 14:19:38.184421 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:38.184463 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:38.185580 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:38.185903 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:38.185907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:38.186240 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:38.186475 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.186480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.187192 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.187193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.188324 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.188326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.188744 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.188745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.188870 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.189455 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:38.189463 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:38.190883 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:38.192606 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:38.192612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:38.192615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:38.192848 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:38.193058 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:38.193058 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:38.193059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:38.193165 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:38.193423 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:38.193429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:38.193573 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:19:38.193690 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:19:38.193852 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260620 14:19:38.194657 8753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45111
I20260620 14:19:38.194765 8753 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:38.194949 8753 heartbeater.cc:511] Master 127.0.98.62:45111 requested a full tablet report, sending...
I20260620 14:19:38.195505 8562 ts_manager.cc:194] Registered new tserver with Master: cabe73b42a5e4aca83eeef3b476a69ae (127.0.98.1:38567)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:19:38.196305 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:38.210920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45111
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:38.292532 8758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:38.292788 8758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:38.292829 8758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:38.294024 8758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:38.294102 8758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:38.294147 8758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:38.295693 8758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:38.295979 8758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:38.296216 8758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:38.301800 8764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.301837 8763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.301848 8766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:38.302119 8758 server_base.cc:1034] running on GCE node
I20260620 14:19:38.302366 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:38.302667 8758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:38.303858 8758 hybrid_clock.cc:648] HybridClock initialized: now 1781965178303855 us; error 62 us; skew 500 ppm
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:38.306718 8758 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:38.307184 8758 webserver.cc:460] Webserver started at http://127.0.98.2:39125/ using document root <none> and password file <none>
I20260620 14:19:38.307382 8758 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:38.307425 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:38.307513 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:38.308575 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "71eba6c67c224e00a78669f3a42d8a93"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "718584b1df09eb22b002ccc9cc850730"
server_key_iv: "18016d3fa88251bd53d382d23fca4037"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.308900 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "71eba6c67c224e00a78669f3a42d8a93"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "718584b1df09eb22b002ccc9cc850730"
server_key_iv: "18016d3fa88251bd53d382d23fca4037"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.310209 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:38.310945 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.311152 8758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:38.311239 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "71eba6c67c224e00a78669f3a42d8a93"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "718584b1df09eb22b002ccc9cc850730"
server_key_iv: "18016d3fa88251bd53d382d23fca4037"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.311295 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:38.326121 8758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:38.328613 8758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:38.328733 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:38.328927 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:38.329208 8758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:38.329241 8758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.329263 8758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:38.329276 8758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.334896 8758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39891
I20260620 14:19:38.334977 8886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39891 every 8 connection(s)
I20260620 14:19:38.335299 8758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:38.335614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 8758
I20260620 14:19:38.335726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:38.335932 392 external_mini_cluster.cc:1428] Setting key 5bafae9bf523c1089a28e6e3e6af2d1a
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:19:38.345913 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.336141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46381 (local address 127.0.98.62:45111)
0620 14:19:38.336213 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:38.336215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:38.336453 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:19:38.337595 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:38.337596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:38.337609 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:38.337684 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:38.338276 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.338754 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.340280 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.340487 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.340546 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.343001 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:38.343003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:38.343004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:38.343012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:38.344524 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.344942 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.344943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.344944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.345007 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.345266 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.345267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.345267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.345385 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:38.345463 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:38.345734 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:19:38.345817 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:19:38.346131 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.336090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:46381)
0620 14:19:38.336324 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:38.336372 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:38.337478 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:38.337755 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:38.337759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:38.338007 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:38.338198 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.338201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.338860 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.338861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.340193 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.340196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.340571 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.340573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.340740 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.341251 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:38.341261 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:38.342898 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:38.344627 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:38.344634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:38.344636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:38.344863 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:38.345077 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:38.345078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:38.345078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:38.345192 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:38.345454 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:38.345460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:38.345605 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 14:19:38.345714 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:19:38.345880 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260620 14:19:38.346686 8887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45111
I20260620 14:19:38.346798 8887 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:38.347033 8887 heartbeater.cc:511] Master 127.0.98.62:45111 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:19:38.347481 8562 ts_manager.cc:194] Registered new tserver with Master: 71eba6c67c224e00a78669f3a42d8a93 (127.0.98.2:39891)
I20260620 14:19:38.348065 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:38.361954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45111
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:38.443436 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:38.443660 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:38.443682 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:38.444773 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:38.444809 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:38.444835 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:38.446332 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:38.446516 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:38.446780 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:38.452100 8900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.452060 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.452059 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:38.452371 8892 server_base.cc:1034] running on GCE node
I20260620 14:19:38.452630 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:38.452876 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:38.454034 8892 hybrid_clock.cc:648] HybridClock initialized: now 1781965178454027 us; error 42 us; skew 500 ppm
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:38.456633 8892 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:38.457005 8892 webserver.cc:460] Webserver started at http://127.0.98.3:33771/ using document root <none> and password file <none>
I20260620 14:19:38.457197 8892 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:38.457237 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:38.457324 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:38.458284 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ab4de6ce2b643b2a78ddd50c60b4647"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "13ee680a3f7412a2b008c202a0ac91b8"
server_key_iv: "bdd26f754f80b96dd6af6b0cd989dff6"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.458570 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ab4de6ce2b643b2a78ddd50c60b4647"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "13ee680a3f7412a2b008c202a0ac91b8"
server_key_iv: "bdd26f754f80b96dd6af6b0cd989dff6"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.459831 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:38.460496 8907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.460645 8892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:38.460738 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "2ab4de6ce2b643b2a78ddd50c60b4647"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "13ee680a3f7412a2b008c202a0ac91b8"
server_key_iv: "bdd26f754f80b96dd6af6b0cd989dff6"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.460793 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:38.482997 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:38.484987 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:38.485105 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:38.485296 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:38.485617 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:38.485652 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.485695 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:38.485723 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.491186 8892 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34713
I20260620 14:19:38.491289 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34713 every 8 connection(s)
I20260620 14:19:38.491590 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:38.496938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 8892
I20260620 14:19:38.497047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:38.497255 392 external_mini_cluster.cc:1428] Setting key 39c44220155e38889a22e8288a86bb92
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:38.501204 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.492338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54391 (local address 127.0.98.62:45111)
0620 14:19:38.492440 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:38.492442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:38.492579 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:19:38.493504 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:38.493506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:38.493516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:38.493583 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:38.494153 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.494642 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.495689 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.495903 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.495967 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.498493 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:38.498496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:38.498497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:38.498508 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:38.500001 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.500344 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.500344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.500345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.500393 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:38.500610 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:38.500610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:38.500610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:38.500707 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:38.500763 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:38.501069 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 14:19:38.501129 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:19:38.501384 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.492289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:54391)
0620 14:19:38.492456 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:38.492491 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:38.493403 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:38.493642 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:38.493646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:38.493892 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:38.494077 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.494081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.494721 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.494721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.495517 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.495519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.495980 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.495980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.496079 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.496673 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:38.496681 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:38.498420 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:38.500067 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:38.500071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:38.500073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:38.500281 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:38.500449 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:38.500450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:38.500450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:38.500539 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:38.500822 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:38.500828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:38.500945 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 14:19:38.501042 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:19:38.501172 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
I20260620 14:19:38.501787 9021 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45111
I20260620 14:19:38.501907 9021 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:38.502091 9021 heartbeater.cc:511] Master 127.0.98.62:45111 requested a full tablet report, sending...
I20260620 14:19:38.502444 8562 ts_manager.cc:194] Registered new tserver with Master: 2ab4de6ce2b643b2a78ddd50c60b4647 (127.0.98.3:34713)
I20260620 14:19:38.502820 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54391
I20260620 14:19:38.509004 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:19:38.581867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 8624
I20260620 14:19:38.586690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 8758
I20260620 14:19:38.591012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 8892
I20260620 14:19:38.595177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 8530
I20260620 14:19:38.600284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45111
--webserver_interface=127.0.98.62
--webserver_port=36349
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:38.681828 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:38.682057 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:38.682080 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:38.683280 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:38.683328 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:38.683343 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:38.683355 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:38.683367 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:38.684927 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:38.685194 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:38.685443 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:38.690263 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.690269 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.690269 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:38.690625 9030 server_base.cc:1034] running on GCE node
I20260620 14:19:38.690925 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:38.691205 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:38.692344 9030 hybrid_clock.cc:648] HybridClock initialized: now 1781965178692328 us; error 26 us; skew 500 ppm
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:38.694977 9030 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:38.695514 9030 webserver.cc:460] Webserver started at http://127.0.98.62:36349/ using document root <none> and password file <none>
I20260620 14:19:38.695765 9030 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:38.695851 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:38.697638 9030 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:38.698407 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.698678 9030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:38.698788 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "570f4d960a4b421c8241992f6c8f0a08"
format_stamp: "Formatted at 2026-06-20 14:19:37 on dist-test-slave-h6n5"
server_key: "ac516e90131edd288e394787b2fbcf35"
server_key_iv: "6f49bd90df9600267b92cf83b18c9ad8"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.699193 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:38.717810 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:38.720280 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:38.720453 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:38.724500 9030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45111
I20260620 14:19:38.724539 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45111 every 8 connection(s)
I20260620 14:19:38.725128 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:38.726408 9098 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:19:38.727113 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Bootstrap starting.
I20260620 14:19:38.735097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9030
I20260620 14:19:38.735397 9098 log.cc:824] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:38.735442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38567
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45111
--webserver_port=42675
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:19:38.737397 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:19:38.737684 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Bootstrap complete.
I20260620 14:19:38.740027 9098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:38.740247 9098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570f4d960a4b421c8241992f6c8f0a08, State: Initialized, Role: FOLLOWER
I20260620 14:19:38.740410 9098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:38.740489 9098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:38.740541 9098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:38.740590 9098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:19:38.741562 9098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:38.741648 9098 leader_election.cc:302] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 570f4d960a4b421c8241992f6c8f0a08; no voters:
I20260620 14:19:38.741838 9098 leader_election.cc:288] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:19:38.741884 9102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:19:38.741999 9102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [term 2 LEADER]: Becoming Leader. State: Replica: 570f4d960a4b421c8241992f6c8f0a08, State: Running, Role: LEADER
I20260620 14:19:38.742085 9102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } }
I20260620 14:19:38.742116 9098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:38.742385 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "570f4d960a4b421c8241992f6c8f0a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } } }
I20260620 14:19:38.742651 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:38.742573 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 570f4d960a4b421c8241992f6c8f0a08. Latest consensus state: current_term: 2 leader_uuid: "570f4d960a4b421c8241992f6c8f0a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570f4d960a4b421c8241992f6c8f0a08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45111 } } }
I20260620 14:19:38.742929 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:38.742972 9112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:38.743546 9112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:38.743733 9112 catalog_manager.cc:1259] Loaded cluster ID: 84220fc7c3ec491599131fc7a7e29a3e
I20260620 14:19:38.743927 9112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:38.744724 9112 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:38.744908 9112 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 570f4d960a4b421c8241992f6c8f0a08: Loaded TSK: 0
I20260620 14:19:38.745136 9112 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:19:38.830235 9101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:38.830430 9101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:38.830453 9101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:38.831704 9101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:38.831745 9101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:38.831772 9101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:38.833195 9101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42675
--tserver_master_addrs=127.0.98.62:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:38.833420 9101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:38.833631 9101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:38.838749 9124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.838721 9127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.838733 9125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:38.839017 9101 server_base.cc:1034] running on GCE node
I20260620 14:19:38.839214 9101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:38.839448 9101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:38.840621 9101 hybrid_clock.cc:648] HybridClock initialized: now 1781965178840611 us; error 49 us; skew 500 ppm
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:38.843492 9101 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:19:38.844023 9101 webserver.cc:460] Webserver started at http://127.0.98.1:42675/ using document root <none> and password file <none>
I20260620 14:19:38.844285 9101 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:38.844357 9101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:38.846022 9101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:19:38.846668 9134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.846879 9101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:19:38.846971 9101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "cabe73b42a5e4aca83eeef3b476a69ae"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "3abf8c895025db6dbc357f88b2b5ec8f"
server_key_iv: "2a4d37fe8a9005a7d316c6ccaacf8546"
server_key_version: "encryptionkey@0"
I20260620 14:19:38.847481 9101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:38.884713 9101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:38.887455 9101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:38.887637 9101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:38.887892 9101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:38.888216 9101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:38.888255 9101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.888285 9101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:38.888302 9101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:38.894934 9101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38567
I20260620 14:19:38.895010 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38567 every 8 connection(s)
I20260620 14:19:38.895458 9101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:38.901068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9101
I20260620 14:19:38.901383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39891
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45111
--webserver_port=39125
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:38.904265 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.896287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:45699)
0620 14:19:38.896480 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:38.896529 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:38.897047 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:38.897531 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:38.897535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:38.898122 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:38.898309 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.898314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.899422 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.899424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.900276 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.900278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.900886 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.900887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.901015 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.901631 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:38.901641 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:38.904122 (+ 2481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
W20260620 14:19:38.904457 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:38.896345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45699 (local address 127.0.98.62:45111)
0620 14:19:38.896563 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:38.896573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:38.896611 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:19:38.897281 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:38.897286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:38.897339 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:38.897482 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:38.898431 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.899314 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.900376 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.900779 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.900900 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.904278 (+ 3378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45699: BlockingRecv error: recv got EOF from 127.0.98.1:45699 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":102,"threads_started":1}
W20260620 14:19:38.904500 9248 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45111 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45111: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:38.909139 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.904706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:55399)
0620 14:19:38.904798 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:38.904810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:38.904866 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:38.904987 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:38.904988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:38.905167 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:38.905219 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.905221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.905851 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.905852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.906366 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.906367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.906743 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.906743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.906806 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.907307 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:38.907311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:38.909050 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:19:38.909191 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:38.904760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55399 (local address 127.0.98.62:45111)
0620 14:19:38.904817 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:38.904819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:38.904850 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:19:38.904924 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:38.904925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:38.904936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:38.904987 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:38.905301 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.905770 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.906462 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.906662 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.906734 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.909126 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55399: BlockingRecv error: recv got EOF from 127.0.98.1:55399 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:38.913656 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:38.909431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:45595)
0620 14:19:38.909461 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:38.909472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:38.909509 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:38.909814 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:38.909814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:38.909934 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:38.909960 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.909961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.910599 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.910600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.911087 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:38.911088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.911425 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.911426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:38.911474 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.911926 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:38.911930 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:38.913595 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 14:19:38.913769 9248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:19:38.913811 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:38.909659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45595 (local address 127.0.98.62:45111)
0620 14:19:38.909716 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:38.909718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:38.909724 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:19:38.909740 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:38.909740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:38.909748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:38.909812 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:38.910075 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.910520 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.911180 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:38.911355 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:38.911427 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:38.913670 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45595: BlockingRecv error: recv got EOF from 127.0.98.1:45595 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:19:38.984447 9252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:38.984654 9252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:38.984679 9252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:38.985853 9252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:38.985896 9252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:38.985922 9252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:38.987424 9252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39125
--tserver_master_addrs=127.0.98.62:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:38.987709 9252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:38.987946 9252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:38.992801 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.992784 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:38.992784 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:38.993175 9252 server_base.cc:1034] running on GCE node
I20260620 14:19:38.993368 9252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:38.993594 9252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:38.994762 9252 hybrid_clock.cc:648] HybridClock initialized: now 1781965178994738 us; error 34 us; skew 500 ppm
Jun 20 14:19:38 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965178, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:39.000545 9252 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:19:39.000973 9252 webserver.cc:460] Webserver started at http://127.0.98.2:39125/ using document root <none> and password file <none>
I20260620 14:19:39.001159 9252 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:39.001210 9252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:39.002586 9252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:39.003162 9267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:39.003325 9252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:39.003403 9252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "71eba6c67c224e00a78669f3a42d8a93"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "718584b1df09eb22b002ccc9cc850730"
server_key_iv: "18016d3fa88251bd53d382d23fca4037"
server_key_version: "encryptionkey@0"
I20260620 14:19:39.003775 9252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:39.019385 9252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:39.021638 9252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:39.021747 9252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:39.021917 9252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:39.022218 9252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:39.022258 9252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:39.022298 9252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:39.022326 9252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:39.027925 9252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39891
I20260620 14:19:39.027992 9380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39891 every 8 connection(s)
I20260620 14:19:39.028313 9252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:19:39.035611 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:39.029059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41031 (local address 127.0.98.62:45111)
0620 14:19:39.029125 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.029127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.029338 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:19:39.029865 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.029867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.029881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.029935 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.030869 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.031371 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.032375 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.032589 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.032674 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.035526 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41031: BlockingRecv error: recv got EOF from 127.0.98.2:41031 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:19:39.035677 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.029050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:41031)
0620 14:19:39.029220 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:39.029253 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:39.029760 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:39.030004 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:39.030008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:39.030580 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:39.030763 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.030768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.031466 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.031467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.032283 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.032285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.032682 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.032683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.032822 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.033252 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:39.033260 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:39.035455 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
W20260620 14:19:39.035885 9381 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45111 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45111: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 14:19:39.036934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9252
I20260620 14:19:39.037225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34713
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45111
--webserver_port=33771
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:39.040831 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.036116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:53387)
0620 14:19:39.036168 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:39.036180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:39.036227 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:39.036339 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:39.036339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:39.036444 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:39.036486 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.036487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.037310 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.037312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.037857 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.037859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.038344 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.038345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.038418 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.038789 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:39.038794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:39.040734 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:19:39.040890 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:39.036173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53387 (local address 127.0.98.62:45111)
0620 14:19:39.036231 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.036233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.036240 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:39.036278 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.036279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.036287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.036336 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.036540 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.037203 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.037936 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.038252 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.038323 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.040823 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53387: BlockingRecv error: recv got EOF from 127.0.98.2:53387 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:39.045818 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.041095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:56195)
0620 14:19:39.041137 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:39.041147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:39.041182 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:39.041370 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:39.041391 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:39.041537 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:39.041575 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.041577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.042181 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.042182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.043088 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.043091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.043426 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.043427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.043484 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.044008 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:39.044012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:39.045739 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:19:39.045867 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:39.041144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56195 (local address 127.0.98.62:45111)
0620 14:19:39.041260 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.041261 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.041268 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:39.041286 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.041287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.041294 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.041343 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.041644 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.042103 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.043164 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.043335 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.043427 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.045805 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56195: BlockingRecv error: recv got EOF from 127.0.98.2:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 14:19:39.045984 9381 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:19:39.119022 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:39.119277 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:39.119316 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:39.120544 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:39.120618 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:39.120666 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:39.122107 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33771
--tserver_master_addrs=127.0.98.62:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:39.122372 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:39.122610 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:39.127609 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:39.127539 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:39.127540 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:39.127801 9384 server_base.cc:1034] running on GCE node
I20260620 14:19:39.127996 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:39.128196 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:39.129370 9384 hybrid_clock.cc:648] HybridClock initialized: now 1781965179129357 us; error 30 us; skew 500 ppm
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:39.132006 9384 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:19:39.132495 9384 webserver.cc:460] Webserver started at http://127.0.98.3:33771/ using document root <none> and password file <none>
I20260620 14:19:39.132700 9384 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:39.132756 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:39.134115 9384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:39.134816 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:39.134990 9384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:39.135079 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "2ab4de6ce2b643b2a78ddd50c60b4647"
format_stamp: "Formatted at 2026-06-20 14:19:38 on dist-test-slave-h6n5"
server_key: "13ee680a3f7412a2b008c202a0ac91b8"
server_key_iv: "bdd26f754f80b96dd6af6b0cd989dff6"
server_key_version: "encryptionkey@0"
I20260620 14:19:39.135402 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:39.148881 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:39.151181 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:39.151291 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:39.151518 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:39.151835 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:39.151868 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:39.151890 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:39.151902 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:39.157541 9384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34713
I20260620 14:19:39.157588 9512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34713 every 8 connection(s)
I20260620 14:19:39.157910 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:39.163049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9384
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:19:39.165591 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:39.158617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47619 (local address 127.0.98.62:45111)
0620 14:19:39.158688 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.158689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.158852 (+ 163us) server_negotiation.cc:407] Connection header received
0620 14:19:39.159422 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.159423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.159434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.159488 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.160643 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.161125 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.162107 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.162290 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.162347 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.165525 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47619: BlockingRecv error: recv got EOF from 127.0.98.3:47619 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:19:39.165661 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.158607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:47619)
0620 14:19:39.158752 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:39.158787 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:39.159301 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:39.159704 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:39.159708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:39.160338 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:39.160528 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.160533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.161217 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.161218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.162024 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.162026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.162366 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.162367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.162464 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.163010 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:39.163018 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:39.165465 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
W20260620 14:19:39.165895 9513 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45111 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45111: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:39.170367 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.166159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:55831)
0620 14:19:39.166201 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:39.166216 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:39.166264 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:39.166431 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:39.166432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:39.166578 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:39.166643 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.166645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.167238 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.167240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.167789 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.167791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.168152 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.168153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.168217 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.168579 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:39.168583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:39.170291 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:19:39.170410 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:39.166222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55831 (local address 127.0.98.62:45111)
0620 14:19:39.166278 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.166280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.166288 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:39.166339 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.166340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.166355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.166426 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.166732 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.167164 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.167877 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.168054 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.168137 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.170349 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55831: BlockingRecv error: recv got EOF from 127.0.98.3:55831 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:19:39.171373 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.163612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.0.98.62:45111)
0620 14:19:39.163763 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.163766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.163775 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:39.163795 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.163796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.163808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.163879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.164593 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.165137 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.166038 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.166309 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.166388 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.167712 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:39.167717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:39.167717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:39.167728 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:39.169510 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:39.169920 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:39.169923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:39.169923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:39.170067 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:39.170539 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:39.170539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:39.170540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:39.170893 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:39.171009 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:39.171175 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:19:39.171284 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:39.174602 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.170629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:57367)
0620 14:19:39.170681 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:39.170692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:39.170725 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:39.170868 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:39.170868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:39.171008 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:39.171038 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.171040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.171863 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.171864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.172431 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.172433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.172724 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.172725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.172757 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.173153 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:39.173158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:39.174560 (+ 1402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:19:39.174666 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:39.170709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57367 (local address 127.0.98.62:45111)
0620 14:19:39.170760 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.170762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.170769 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:19:39.170787 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.170787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.170796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.170853 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.171116 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.171791 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.172494 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.172666 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.172715 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.174632 (+ 1917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57367: BlockingRecv error: recv got EOF from 127.0.98.3:57367 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 14:19:39.174844 9513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:39 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:39.920042 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:39.914189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:40395)
0620 14:19:39.914307 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:39.914326 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:39.914376 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:39.914603 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:39.914604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:39.914871 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:39.914948 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.914972 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.915915 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.915917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.916592 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:39.916596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.917097 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.917098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:39.917154 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.917676 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:39.917681 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:39.919948 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260620 14:19:39.920167 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:39.914242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40395 (local address 127.0.98.62:45111)
0620 14:19:39.914413 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:39.914415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:39.914451 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:19:39.914484 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:39.914485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:39.914500 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:39.914589 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:39.915070 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.915835 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.916693 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:39.916960 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:39.917029 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:39.920054 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40395: BlockingRecv error: recv got EOF from 127.0.98.1:40395 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
Jun 20 14:19:40 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:40 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:40.050870 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:40.046370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:55289)
0620 14:19:40.046508 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:40.046518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:40.046573 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:40.046678 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:40.046679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:40.046828 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:40.046875 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:40.046877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.047479 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.047479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:40.048052 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:40.048054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.048405 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.048406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:40.048445 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:40.048895 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:40.048900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:40.050763 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
W20260620 14:19:40.050916 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:40.046421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55289 (local address 127.0.98.62:45111)
0620 14:19:40.046502 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:40.046504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:40.046564 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:19:40.046615 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:40.046616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:40.046626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:40.046672 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:40.046931 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.047411 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.048141 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.048334 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.048398 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:40.050847 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55289: BlockingRecv error: recv got EOF from 127.0.98.2:55289 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:19:40 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:40 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:40.179852 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:40.175249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:57817)
0620 14:19:40.175406 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:40.175425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:40.175484 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:40.175638 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:40.175639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:40.175798 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:40.175846 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:40.175847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.176665 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.176666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:40.177191 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:40.177193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.177560 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.177560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:40.177621 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:40.178056 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:40.178061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:40.179785 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
W20260620 14:19:40.179944 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:40.175361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57817 (local address 127.0.98.62:45111)
0620 14:19:40.175426 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:40.175428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:40.175485 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:19:40.175527 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:40.175528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:40.175540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:40.175618 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:40.175918 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.176598 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.177301 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.177485 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.177550 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:40.179876 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57817: BlockingRecv error: recv got EOF from 127.0.98.3:57817 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:19:40 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:40 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:40.924867 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:40.920757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:44545)
0620 14:19:40.920871 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:40.920888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:40.920933 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:40.921054 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:40.921055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:40.921222 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:40.921270 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:40.921271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.921877 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.921878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:40.922385 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:40.922386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.922690 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.922690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:40.922726 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:40.923151 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:40.923155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:40.924810 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
W20260620 14:19:40.924970 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:40.920823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44545 (local address 127.0.98.62:45111)
0620 14:19:40.920949 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:40.920950 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:40.920958 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:40.920982 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:40.920982 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:40.920992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:40.921050 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:40.921331 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.921810 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.922451 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:40.922628 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:40.922675 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:40.924897 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44545: BlockingRecv error: recv got EOF from 127.0.98.1:44545 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
Jun 20 14:19:41 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:41 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:41.055907 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:41.051360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:43031)
0620 14:19:41.051514 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:41.051526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:41.051630 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:41.051804 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:41.051805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:41.051982 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:41.052029 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:41.052031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.052768 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.052769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:41.053327 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:41.053330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.053641 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.053641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:41.053682 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:41.054123 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:41.054128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:41.055840 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
W20260620 14:19:41.055980 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:41.051489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43031 (local address 127.0.98.62:45111)
0620 14:19:41.051587 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:41.051589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:41.051599 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:19:41.051710 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:41.051711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:41.051723 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:41.051800 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:41.052112 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.052667 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.053401 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.053577 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.053630 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:41.055931 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43031: BlockingRecv error: recv got EOF from 127.0.98.2:43031 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:19:41 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:41 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:41.185063 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:41.180380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:34653)
0620 14:19:41.180582 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:41.180594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:41.180647 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:41.180926 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:41.180927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:41.181096 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:41.181146 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:41.181148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.181806 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.181806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:41.182327 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:41.182329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.182683 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.182684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:41.182729 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:41.183238 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:41.183243 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:41.184990 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
W20260620 14:19:41.185192 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:41.180604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34653 (local address 127.0.98.62:45111)
0620 14:19:41.180769 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:41.180773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:41.180787 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:19:41.180809 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:41.180810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:41.180819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:41.180870 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:41.181227 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.181725 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.182408 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.182609 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.182664 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:41.185089 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34653: BlockingRecv error: recv got EOF from 127.0.98.3:34653 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Jun 20 14:19:41 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:41 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:41.931821 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:41.925357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:41637)
0620 14:19:41.925482 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:41.925494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:41.925550 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:41.925707 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:41.925708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:41.925865 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:41.925916 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:41.925918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.926992 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.926993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:41.928123 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:41.928126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.928680 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.928681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:41.928734 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:41.929397 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:41.929403 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:41.931710 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260620 14:19:41.931996 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:41.925421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41637 (local address 127.0.98.62:45111)
0620 14:19:41.925540 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:41.925542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:41.925553 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:41.925608 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:41.925609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:41.925623 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:41.925700 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:41.925995 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.926887 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.928237 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:41.928571 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:41.928643 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:41.931821 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41637: BlockingRecv error: recv got EOF from 127.0.98.1:41637 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
Jun 20 14:19:42 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:42 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:42.061295 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:42.056416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:60211)
0620 14:19:42.056644 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:42.056657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:42.056759 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:42.056926 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:42.056927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:42.057130 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:42.057179 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:42.057181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.057855 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.057856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:42.058390 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:42.058392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.058754 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.058755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:42.058814 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:42.059296 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:42.059300 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:42.061200 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
W20260620 14:19:42.061352 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:42.056530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60211 (local address 127.0.98.62:45111)
0620 14:19:42.056629 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:42.056632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:42.056739 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:19:42.056832 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:42.056834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:42.056852 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:42.056920 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:42.057266 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.057769 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.058486 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.058682 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.058755 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:42.061287 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60211: BlockingRecv error: recv got EOF from 127.0.98.2:60211 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 14:19:42 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:42 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:42.190531 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:42.185576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:55905)
0620 14:19:42.185760 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:42.185773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:42.185871 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:42.186040 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:42.186041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:42.186318 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:42.186370 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:42.186373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.187099 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.187100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:42.187702 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:42.187705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.188073 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.188074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:42.188142 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:42.188635 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:42.188640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:42.190463 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
W20260620 14:19:42.190658 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:42.185691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55905 (local address 127.0.98.62:45111)
0620 14:19:42.185770 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:42.185773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:42.185852 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:19:42.185929 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:42.185930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:42.185944 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:42.186035 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:42.186492 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.187003 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.187783 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.187987 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.188073 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:42.190554 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55905: BlockingRecv error: recv got EOF from 127.0.98.3:55905 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:19:42 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:42 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:42.937073 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:42.932376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:43771)
0620 14:19:42.932515 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:42.932526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:42.932593 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:42.932776 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:42.932777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:42.932939 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:42.932990 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:42.932992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.933677 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.933678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:42.934232 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:42.934234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.934617 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.934618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:42.934658 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:42.935179 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:42.935183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:42.937011 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260620 14:19:42.937271 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:42.932400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43771 (local address 127.0.98.62:45111)
0620 14:19:42.932540 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:42.932542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:42.932576 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:19:42.932636 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:42.932636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:42.932646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:42.932707 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:42.933084 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.933603 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.934336 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:42.934541 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:42.934607 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:42.937151 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43771: BlockingRecv error: recv got EOF from 127.0.98.1:43771 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
Jun 20 14:19:43 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:43 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:43.066900 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:43.061884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:58927)
0620 14:19:43.062073 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:43.062084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:43.062176 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:43.062348 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:43.062349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:43.062562 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:43.062624 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:43.062627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.063383 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.063384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:43.063999 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:43.064001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.064368 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.064370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:43.064432 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:43.064952 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:43.064957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:43.066805 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
W20260620 14:19:43.066944 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:43.061980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58927 (local address 127.0.98.62:45111)
0620 14:19:43.062049 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:43.062052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:43.062152 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:19:43.062249 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:43.062250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:43.062263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:43.062342 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:43.062735 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.063290 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.064086 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.064282 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.064367 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:43.066882 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58927: BlockingRecv error: recv got EOF from 127.0.98.2:58927 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 14:19:43 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:43 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:43.195593 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:43.191043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:59643)
0620 14:19:43.191274 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:43.191285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:43.191352 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:43.191474 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:43.191474 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:43.191643 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:43.191690 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:43.191691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.192305 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.192306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:43.192811 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:43.192812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.193117 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.193118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:43.193154 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:43.193777 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:43.193783 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:43.195491 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
W20260620 14:19:43.195739 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:43.191198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59643 (local address 127.0.98.62:45111)
0620 14:19:43.191282 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:43.191284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:43.191347 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:19:43.191412 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:43.191413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:43.191425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:43.191462 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:43.191752 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.192233 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.192881 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.193053 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.193103 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:43.195656 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59643: BlockingRecv error: recv got EOF from 127.0.98.3:59643 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:19:43 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:43 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:43.942437 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:43.937634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:36783)
0620 14:19:43.937785 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:43.937799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:43.937876 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:43.938018 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:43.938018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:43.938263 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:43.938313 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:43.938315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.939065 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.939066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:43.939702 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:43.939704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.940023 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.940024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:43.940095 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:43.940545 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:43.940550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:43.942376 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
W20260620 14:19:43.942716 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:43.937692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36783 (local address 127.0.98.62:45111)
0620 14:19:43.937826 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:43.937827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:43.937859 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:19:43.937931 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:43.937932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:43.937943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:43.937999 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:43.938376 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.938948 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.939768 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:43.939959 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:43.940013 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:43.942595 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36783: BlockingRecv error: recv got EOF from 127.0.98.1:36783 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
Jun 20 14:19:44 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:44 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:44.075796 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:44.067359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:47181)
0620 14:19:44.067507 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:44.067517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:44.067636 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:44.067794 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:44.067795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:44.067952 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:44.068002 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:44.068004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.068610 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.068611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:44.069154 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:44.069156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.069459 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.069459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:44.069497 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:44.073836 (+ 4339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:44.073841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:44.075685 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260620 14:19:44.075839 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:44.067466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47181 (local address 127.0.98.62:45111)
0620 14:19:44.067527 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:44.067529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:44.067603 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:19:44.067704 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:44.067705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:44.067714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:44.067774 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:44.068064 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.068544 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.069223 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.069396 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.069445 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:44.075774 (+ 6329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47181: BlockingRecv error: recv got EOF from 127.0.98.2:47181 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 14:19:44 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:44 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:44.200696 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:44.196107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:49241)
0620 14:19:44.196269 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:44.196280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:44.196344 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:44.196468 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:44.196469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:44.196624 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:44.196674 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:44.196675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.197366 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.197367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:44.197889 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:44.197890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.198248 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.198248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:44.198306 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:44.198771 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:44.198775 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:44.200639 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
W20260620 14:19:44.200811 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:44.196202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49241 (local address 127.0.98.62:45111)
0620 14:19:44.196284 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:44.196286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:44.196327 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:19:44.196388 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:44.196389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:44.196400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:44.196447 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:44.196769 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.197274 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.197983 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.198172 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.198250 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:44.200733 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49241: BlockingRecv error: recv got EOF from 127.0.98.3:49241 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:19:44 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:44 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:44.947916 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:44.943016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:56889)
0620 14:19:44.943188 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:44.943200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:44.943265 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:44.943393 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:44.943393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:44.943612 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:44.943679 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:44.943681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.944304 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.944305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:44.944847 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:44.944848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.945192 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.945193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:44.945246 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:44.945806 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:44.945811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:44.947843 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
W20260620 14:19:44.948135 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:44.943079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56889 (local address 127.0.98.62:45111)
0620 14:19:44.943211 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:44.943213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:44.943269 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:19:44.943311 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:44.943311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:44.943322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:44.943389 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:44.943740 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.944231 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.944916 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:44.945108 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:44.945163 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:44.948064 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56889: BlockingRecv error: recv got EOF from 127.0.98.1:56889 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
Jun 20 14:19:45 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:45 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:45.081225 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:45.076302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:35901)
0620 14:19:45.076570 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:45.076582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:45.076657 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:45.076787 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:45.076788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:45.076973 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:45.077020 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:45.077023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.077693 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.077695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:45.078272 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:45.078275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.078658 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.078660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:45.078719 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:45.079282 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:45.079287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:45.081112 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":191,"threads_started":1}
W20260620 14:19:45.081301 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:45.076480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35901 (local address 127.0.98.62:45111)
0620 14:19:45.076557 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:45.076560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:45.076654 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:19:45.076716 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:45.076718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:45.076730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:45.076783 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:45.077103 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.077605 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.078366 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.078571 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.078656 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:45.081220 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35901: BlockingRecv error: recv got EOF from 127.0.98.2:35901 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:19:45 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:45 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:45.205900 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:45.201196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:43443)
0620 14:19:45.201352 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:45.201363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:45.201424 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:45.201556 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:45.201556 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:45.201734 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:45.201788 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:45.201790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.202465 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.202466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:45.203021 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:45.203023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.203366 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.203366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:45.203406 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:45.203927 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:45.203931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:45.205790 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
W20260620 14:19:45.205929 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:45.201226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43443 (local address 127.0.98.62:45111)
0620 14:19:45.201296 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:45.201298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:45.201446 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:19:45.201473 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:45.201474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:45.201486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:45.201542 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:45.201882 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.202396 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.203097 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.203297 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.203352 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:45.205865 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43443: BlockingRecv error: recv got EOF from 127.0.98.3:43443 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 14:19:45 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:45 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:45.953066 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:45.948376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:34881)
0620 14:19:45.948531 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:45.948542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:45.948594 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:45.948735 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:45.948735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:45.948953 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:45.949016 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:45.949018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.949672 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.949673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:45.950198 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:45.950199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.950519 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.950520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:45.950556 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:45.950986 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:45.950990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:45.952968 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
W20260620 14:19:45.953122 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:45.948479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34881 (local address 127.0.98.62:45111)
0620 14:19:45.948584 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:45.948586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:45.948594 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:45.948639 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:45.948640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:45.948650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:45.948706 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:45.949083 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.949597 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.950265 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:45.950458 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:45.950507 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:45.953049 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34881: BlockingRecv error: recv got EOF from 127.0.98.1:34881 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":55,"threads_started":1}
Jun 20 14:19:46 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:46 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:46.086817 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:46.081776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:35619)
0620 14:19:46.081898 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:46.081916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:46.081964 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:46.082221 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:46.082222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:46.082395 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:46.082445 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:46.082446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.083152 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.083153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:46.083727 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:46.083728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.084110 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.084111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:46.084163 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:46.084611 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:46.084616 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:46.086722 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260620 14:19:46.086866 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:46.081845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35619 (local address 127.0.98.62:45111)
0620 14:19:46.081936 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:46.081939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:46.081955 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:19:46.082023 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:46.082024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:46.082034 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:46.082090 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:46.082533 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.083037 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.083818 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.084015 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.084084 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:46.086800 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35619: BlockingRecv error: recv got EOF from 127.0.98.2:35619 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 14:19:46 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:46 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:46.210794 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:46.206360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:49933)
0620 14:19:46.206523 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:46.206538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:46.206604 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:46.206768 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:46.206768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:46.206926 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:46.206975 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:46.206977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.207691 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.207692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:46.208216 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:46.208218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.208577 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.208578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:46.208619 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:46.209055 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:46.209059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:46.210733 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260620 14:19:46.210886 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:46.206477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49933 (local address 127.0.98.62:45111)
0620 14:19:46.206540 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:46.206542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:46.206609 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:19:46.206681 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:46.206682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:46.206693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:46.206750 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:46.207049 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.207601 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.208316 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.208498 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.208567 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:46.210821 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49933: BlockingRecv error: recv got EOF from 127.0.98.3:49933 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:19:46 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:46 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:46.958814 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:46.953556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:59575)
0620 14:19:46.953705 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:46.953716 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:46.953775 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:46.954127 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:46.954128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:46.954302 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:46.954351 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:46.954353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.955109 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.955110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:46.955948 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:46.955950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.956368 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.956370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:46.956435 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:46.956957 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:46.956964 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:46.958711 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
W20260620 14:19:46.958871 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:46.953723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59575 (local address 127.0.98.62:45111)
0620 14:19:46.953964 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:46.953966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:46.953974 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:46.953999 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:46.953999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:46.954009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:46.954070 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:46.954457 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.954986 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.956037 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:46.956252 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:46.956353 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:46.958801 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59575: BlockingRecv error: recv got EOF from 127.0.98.1:59575 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":183,"threads_started":1}
Jun 20 14:19:47 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:47 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:47.092262 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:47.087507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:43473)
0620 14:19:47.087701 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:47.087712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:47.087772 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:47.087901 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:47.087902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:47.088063 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:47.088115 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:47.088116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.088897 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.088898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:47.089436 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:47.089438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.089767 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.089768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:47.089808 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:47.090352 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:47.090356 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:47.092192 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":130,"threads_started":1}
W20260620 14:19:47.092414 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:47.087576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43473 (local address 127.0.98.62:45111)
0620 14:19:47.087649 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:47.087651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:47.087773 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:19:47.087818 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:47.087819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:47.087829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:47.087882 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:47.088180 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.088821 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.089513 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.089695 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.089754 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:47.092333 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43473: BlockingRecv error: recv got EOF from 127.0.98.2:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:19:47 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:47 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:47.215876 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:47.211301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:45913)
0620 14:19:47.211457 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:47.211467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:47.211528 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:47.211707 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:47.211708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:47.211885 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:47.211934 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:47.211936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.212624 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.212625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:47.213162 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:47.213164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.213488 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.213488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:47.213528 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:47.213969 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:47.213974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:47.215808 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
W20260620 14:19:47.215971 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:47.211363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45913 (local address 127.0.98.62:45111)
0620 14:19:47.211413 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:47.211416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:47.211514 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:19:47.211624 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:47.211625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:47.211640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:47.211690 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:47.212004 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.212552 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.213239 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.213421 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.213474 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:47.215900 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45913: BlockingRecv error: recv got EOF from 127.0.98.3:45913 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 14:19:47 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:47 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:47.963653 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:47.959335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:46405)
0620 14:19:47.959468 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:47.959483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:47.959531 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:47.959692 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:47.959693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:47.959861 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:47.959910 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:47.959911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.960544 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.960544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:47.961054 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:47.961056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.961375 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.961376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:47.961413 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:47.961904 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:47.961908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:47.963596 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
W20260620 14:19:47.963762 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:47.959388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46405 (local address 127.0.98.62:45111)
0620 14:19:47.959521 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:47.959523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:47.959532 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:47.959596 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:47.959597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:47.959614 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:47.959677 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:47.959978 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.960478 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.961128 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:47.961312 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:47.961361 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:47.963690 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46405: BlockingRecv error: recv got EOF from 127.0.98.1:46405 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
Jun 20 14:19:48 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:48 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:48.097955 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:48.092876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:40591)
0620 14:19:48.092998 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:48.093009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:48.093061 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:48.093223 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:48.093223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:48.093433 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:48.093512 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:48.093515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.094311 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.094311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:48.094834 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:48.094836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.095211 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.095211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:48.095248 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:48.095824 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:48.095828 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:48.097858 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260620 14:19:48.098003 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:48.092936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40591 (local address 127.0.98.62:45111)
0620 14:19:48.093029 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:48.093032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:48.093073 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:19:48.093158 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:48.093159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:48.093170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:48.093211 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:48.093600 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.094234 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.094941 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.095138 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.095202 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:48.097939 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40591: BlockingRecv error: recv got EOF from 127.0.98.2:40591 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 14:19:48 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:48 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:48.221076 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:48.216399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:41257)
0620 14:19:48.216517 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:48.216529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:48.216589 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:48.216712 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:48.216713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:48.216878 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:48.216925 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:48.216927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.217590 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.217591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:48.218104 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:48.218105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.218487 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.218488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:48.218542 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:48.219098 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:48.219104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:48.220979 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
W20260620 14:19:48.221117 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:48.216480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41257 (local address 127.0.98.62:45111)
0620 14:19:48.216567 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:48.216569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:48.216580 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:48.216627 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:48.216628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:48.216639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:48.216691 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:48.217020 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.217517 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.218198 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.218394 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.218458 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:48.221057 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41257: BlockingRecv error: recv got EOF from 127.0.98.3:41257 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:19:48 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:48 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:48.968791 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:48.964185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:57607)
0620 14:19:48.964349 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:48.964360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:48.964417 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:48.964557 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:48.964558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:48.964732 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:48.964782 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:48.964784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.965411 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.965412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:48.965942 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:48.965943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.966329 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.966330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:48.966371 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:48.966914 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:48.966919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:48.968732 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
W20260620 14:19:48.968950 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:48.964315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57607 (local address 127.0.98.62:45111)
0620 14:19:48.964438 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:48.964440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:48.964448 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:48.964470 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:48.964471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:48.964480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:48.964544 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:48.964849 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.965344 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.966039 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:48.966259 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:48.966320 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:48.968844 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57607: BlockingRecv error: recv got EOF from 127.0.98.1:57607 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
Jun 20 14:19:49 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:49 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:49.108103 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:49.098492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:49367)
0620 14:19:49.098641 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:49.098654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:49.098718 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:49.098898 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:49.098899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:49.099073 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:49.099136 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:49.099138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.099868 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.099869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:49.100615 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:49.100618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.101097 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.101098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:49.101163 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:49.106077 (+ 4914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:49.106083 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:49.108004 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
W20260620 14:19:49.108157 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:49.098606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49367 (local address 127.0.98.62:45111)
0620 14:19:49.098683 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:49.098685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:49.098699 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:19:49.098805 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:49.098806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:49.098824 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:49.098891 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:49.099225 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.099784 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.100736 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.100963 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.101052 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:49.108092 (+ 7040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49367: BlockingRecv error: recv got EOF from 127.0.98.2:49367 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:19:49 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:49 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:49.226459 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:49.221615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:36625)
0620 14:19:49.221762 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:49.221773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:49.221822 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:49.221957 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:49.221958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:49.222129 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:49.222187 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:49.222188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.222842 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.222843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:49.223390 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:49.223392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.223852 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.223854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:49.223925 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:49.224446 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:49.224451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:49.226399 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20260620 14:19:49.226615 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:49.221727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36625 (local address 127.0.98.62:45111)
0620 14:19:49.221789 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:49.221791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:49.221812 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:19:49.221865 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:49.221866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:49.221877 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:49.221927 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:49.222251 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.222745 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.223486 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.223757 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.223842 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:49.226513 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36625: BlockingRecv error: recv got EOF from 127.0.98.3:36625 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 14:19:49 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:49 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:49.974033 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:49.969387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:56783)
0620 14:19:49.969526 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:49.969536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:49.969621 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:49.969937 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:49.969938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:49.970113 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:49.970163 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:49.970164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.970820 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.970821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:49.971343 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:49.971344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.971694 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.971695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:49.971740 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:49.972179 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:49.972183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:49.973972 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260620 14:19:49.974184 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:49.969487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56783 (local address 127.0.98.62:45111)
0620 14:19:49.969663 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:49.969665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:49.969682 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:19:49.969716 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:49.969722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:49.969733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:49.969853 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:49.970249 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.970750 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.971422 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:49.971632 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:49.971681 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:49.974081 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56783: BlockingRecv error: recv got EOF from 127.0.98.1:56783 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
Jun 20 14:19:50 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:50 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:50.113222 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:50.108582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:45583)
0620 14:19:50.108723 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:50.108739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:50.108796 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:50.108928 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:50.108928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:50.109088 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:50.109137 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:50.109138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.109800 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.109801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:50.110397 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:50.110400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.110758 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.110759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:50.110816 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:50.111258 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:50.111263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:50.113154 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
W20260620 14:19:50.113330 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:50.108683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45583 (local address 127.0.98.62:45111)
0620 14:19:50.108748 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:50.108750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:50.108782 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:19:50.108846 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:50.108847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:50.108858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:50.108909 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:50.109209 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.109716 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.110475 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.110658 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.110710 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:50.113256 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45583: BlockingRecv error: recv got EOF from 127.0.98.2:45583 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:19:50 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:50 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:50.231963 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:50.226957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:44221)
0620 14:19:50.227136 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:50.227147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:50.227217 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:50.227388 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:50.227389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:50.227633 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:50.227688 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:50.227690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.228365 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.228366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:50.228898 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:50.228900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.229248 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.229249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:50.229306 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:50.229823 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:50.229828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:50.231905 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
W20260620 14:19:50.232108 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:50.227045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44221 (local address 127.0.98.62:45111)
0620 14:19:50.227119 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:50.227121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:50.227195 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:19:50.227277 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:50.227278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:50.227297 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:50.227358 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:50.227773 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.228272 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.228982 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.229170 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.229251 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:50.232013 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44221: BlockingRecv error: recv got EOF from 127.0.98.3:44221 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:19:50 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:50 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:50.978922 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:50.974522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:47063)
0620 14:19:50.974699 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:50.974710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:50.974776 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:50.974911 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:50.974912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:50.975071 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:50.975119 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:50.975121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.975787 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.975788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:50.976304 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:50.976306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.976672 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.976673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:50.976724 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:50.977170 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:50.977174 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:50.978862 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
W20260620 14:19:50.979023 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:50.974640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47063 (local address 127.0.98.62:45111)
0620 14:19:50.974770 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:50.974772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:50.974781 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:50.974833 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:50.974834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:50.974844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:50.974898 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:50.975184 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.975712 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.976394 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:50.976583 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:50.976639 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:50.978951 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47063: BlockingRecv error: recv got EOF from 127.0.98.1:47063 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Jun 20 14:19:51 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:51 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:51.118660 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:51.113783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:49955)
0620 14:19:51.113993 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:51.114002 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:51.114088 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:51.114254 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:51.114255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:51.114509 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:51.114572 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:51.114575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.115239 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.115240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:51.115808 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:51.115810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.116147 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.116148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:51.116190 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:51.116664 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:51.116669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:51.118583 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260620 14:19:51.118731 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:51.113916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49955 (local address 127.0.98.62:45111)
0620 14:19:51.113974 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:51.113976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:51.114089 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:19:51.114149 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:51.114150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:51.114164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:51.114226 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:51.114643 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.115144 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.115886 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.116075 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.116132 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:51.118685 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49955: BlockingRecv error: recv got EOF from 127.0.98.2:49955 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 14:19:51 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:51 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:51.237169 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:51.232516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:33153)
0620 14:19:51.232653 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:51.232665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:51.232762 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:51.232888 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:51.232889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:51.233072 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:51.233118 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:51.233120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.233788 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.233789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:51.234332 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:51.234334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.234703 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.234705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:51.234764 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:51.235223 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:51.235227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:51.237110 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
W20260620 14:19:51.237272 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:51.232600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33153 (local address 127.0.98.62:45111)
0620 14:19:51.232653 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:51.232656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:51.232742 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:19:51.232817 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:51.232818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:51.232832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:51.232884 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:51.233200 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.233700 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.234422 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.234617 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.234702 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:51.237204 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33153: BlockingRecv error: recv got EOF from 127.0.98.3:33153 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:19:51 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:51 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:51.984133 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:51.979430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:54705)
0620 14:19:51.979629 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:51.979640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:51.979706 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:51.979844 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:51.979845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:51.980008 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:51.980058 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:51.980060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.980789 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.980791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:51.981379 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:51.981381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.981737 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.981739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:51.981787 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:51.982238 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:51.982242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:51.984066 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
W20260620 14:19:51.984231 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:51.979535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54705 (local address 127.0.98.62:45111)
0620 14:19:51.979657 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:51.979659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:51.979693 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:19:51.979753 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:51.979754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:51.979769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:51.979824 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:51.980135 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.980679 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.981471 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:51.981658 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:51.981711 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:51.984159 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54705: BlockingRecv error: recv got EOF from 127.0.98.1:54705 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
Jun 20 14:19:52 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:52 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:52.123533 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:52.119192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:59155)
0620 14:19:52.119333 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:52.119348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:52.119408 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:52.119533 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:52.119534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:52.119749 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:52.119798 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:52.119799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.120440 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.120441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:52.120957 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:52.120959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.121276 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.121277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:52.121316 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:52.121753 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:52.121758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:52.123473 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
W20260620 14:19:52.123692 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:52.119323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59155 (local address 127.0.98.62:45111)
0620 14:19:52.119400 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:52.119402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:52.119420 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:19:52.119448 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:52.119449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:52.119463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:52.119512 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:52.119882 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.120374 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.121034 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.121213 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.121292 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:52.123613 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59155: BlockingRecv error: recv got EOF from 127.0.98.2:59155 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:19:52 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:52 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:52.242450 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:52.237727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:34119)
0620 14:19:52.237886 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:52.237900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:52.237988 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:52.238140 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:52.238141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:52.238305 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:52.238355 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:52.238357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.239025 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.239026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:52.239653 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:52.239656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.239987 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.239988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:52.240041 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:52.240471 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:52.240477 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:52.242381 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260620 14:19:52.242547 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:52.237872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34119 (local address 127.0.98.62:45111)
0620 14:19:52.237942 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:52.237944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:52.237953 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:52.238055 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:52.238056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:52.238075 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:52.238116 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:52.238424 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.238938 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.239732 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.239917 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.239974 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:52.242476 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34119: BlockingRecv error: recv got EOF from 127.0.98.3:34119 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:19:52 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:52 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:52.989331 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:52.984641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:42825)
0620 14:19:52.984778 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:52.984790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:52.984850 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:52.984997 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:52.984998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:52.985158 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:52.985208 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:52.985210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.986020 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.986021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:52.986532 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:52.986534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.986898 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.986899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:52.986939 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:52.987486 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:52.987490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:52.989273 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
W20260620 14:19:52.989466 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:52.984738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42825 (local address 127.0.98.62:45111)
0620 14:19:52.984864 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:52.984867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:52.984883 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:19:52.984905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:52.984906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:52.984916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:52.984978 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:52.985305 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.985919 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.986616 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:52.986823 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:52.986880 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:52.989384 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42825: BlockingRecv error: recv got EOF from 127.0.98.1:42825 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
Jun 20 14:19:53 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:53 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:53.128885 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:53.124088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:59381)
0620 14:19:53.124273 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:53.124286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:53.124362 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:53.124532 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:53.124533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:53.124717 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:53.124764 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:53.124766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.125467 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.125468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:53.126000 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:53.126001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.126372 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.126373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:53.126431 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:53.126965 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:53.126970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:53.128777 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
W20260620 14:19:53.128914 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:53.124208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59381 (local address 127.0.98.62:45111)
0620 14:19:53.124288 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:53.124290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:53.124342 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:19:53.124432 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:53.124432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:53.124444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:53.124515 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:53.124847 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.125345 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.126088 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.126283 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.126378 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:53.128856 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59381: BlockingRecv error: recv got EOF from 127.0.98.2:59381 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:19:53 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:53 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:53.247750 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:53.243030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.3:55495)
0620 14:19:53.243151 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:53.243168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:53.243249 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:53.243536 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:53.243537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:53.243734 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:53.243778 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:53.243780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.244487 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.244488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:53.245012 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:53.245013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.245407 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.245408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:53.245461 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:53.245904 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:53.245908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:53.247655 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
W20260620 14:19:53.247788 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:53.243046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55495 (local address 127.0.98.62:45111)
0620 14:19:53.243122 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:53.243125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:53.243228 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:19:53.243316 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:53.243317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:53.243331 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:53.243530 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:53.243903 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.244402 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.245114 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.245317 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.245407 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:53.247727 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55495: BlockingRecv error: recv got EOF from 127.0.98.3:55495 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:19:53 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:53 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:53.994899 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:53.989913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.1:54507)
0620 14:19:53.990070 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:53.990081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:53.990136 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:53.990290 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:53.990290 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:53.990465 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:53.990514 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:53.990515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.991375 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.991376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:53.992175 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:53.992177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.992512 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.992512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:53.992551 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:53.993046 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:53.993051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:53.994832 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
W20260620 14:19:53.995096 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:53.990045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54507 (local address 127.0.98.62:45111)
0620 14:19:53.990174 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:53.990176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:53.990184 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:53.990207 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:53.990207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:53.990216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:53.990274 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:53.990629 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.991270 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.992249 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:53.992447 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:53.992503 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:53.995005 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54507: BlockingRecv error: recv got EOF from 127.0.98.1:54507 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
Jun 20 14:19:54 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:19:54 dist-test-slave-h6n5 krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:19:54.136909 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.129468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45111 (local address 127.0.98.2:48345)
0620 14:19:54.129622 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:54.129633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:54.129693 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:54.129818 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:54.129819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:54.129979 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:54.130028 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.130029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.130648 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.130648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.131159 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.131161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.131481 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.131482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.131518 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.135079 (+ 3561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:54.135084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:54.136829 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
W20260620 14:19:54.137013 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:54.129581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48345 (local address 127.0.98.62:45111)
0620 14:19:54.129631 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:54.129633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:54.129699 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:19:54.129740 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:54.129741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:54.129751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:54.129799 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:54.130095 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.130582 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.131227 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.131419 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.131470 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.136919 (+ 5449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48345: BlockingRecv error: recv got EOF from 127.0.98.2:48345 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:54.164187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9101
I20260620 14:19:54.168979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9252
I20260620 14:19:54.173250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9384
I20260620 14:19:54.177492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9030
2026-06-20T14:19:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17501 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:54Z Disabled control of system clock
I20260620 14:19:54.204746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44615
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:19:54.284549 9580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:54.284776 9580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:54.284799 9580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:54.285912 9580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:54.285948 9580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:54.285960 9580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:54.285974 9580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:54.285986 9580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:54.287241 9580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:54.287432 9580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:54.287683 9580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:19:54.292604 9580 server_base.cc:1034] running on GCE node
W20260620 14:19:54.292557 9585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.292573 9586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.292557 9588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:54.293095 9580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:54.293444 9580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:54.294610 9580 hybrid_clock.cc:648] HybridClock initialized: now 1781965194294596 us; error 26 us; skew 500 ppm
I20260620 14:19:54.296034 9580 webserver.cc:460] Webserver started at http://127.0.98.62:36605/ using document root <none> and password file <none>
I20260620 14:19:54.296286 9580 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:54.296366 9580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:54.296504 9580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:54.297624 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "1972ab74d562442696e2c5441012da6e"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "0ef816fae614a3262f6d9833f551e97a"
server_key_iv: "435f8115550a2ccf97681573e95582cd"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.298009 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "1972ab74d562442696e2c5441012da6e"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "0ef816fae614a3262f6d9833f551e97a"
server_key_iv: "435f8115550a2ccf97681573e95582cd"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.299428 9580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:19:54.300205 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.300406 9580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:54.300508 9580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "1972ab74d562442696e2c5441012da6e"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "0ef816fae614a3262f6d9833f551e97a"
server_key_iv: "435f8115550a2ccf97681573e95582cd"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.300596 9580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:54.311195 9580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:54.311486 9580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:54.311681 9580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:54.315276 9580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44615
I20260620 14:19:54.315326 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44615 every 8 connection(s)
I20260620 14:19:54.315742 9580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:54.316423 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:54.318585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9580
I20260620 14:19:54.318682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:54.318929 392 external_mini_cluster.cc:1428] Setting key 24d23cd0cc3e890c0547b219df7bc350
I20260620 14:19:54.319375 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e: Bootstrap starting.
I20260620 14:19:54.320150 9647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:54.320470 9647 log.cc:824] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:54.321256 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e: No bootstrap required, opened a new log
I20260620 14:19:54.323377 9647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972ab74d562442696e2c5441012da6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44615 } }
I20260620 14:19:54.323516 9647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:54.323539 9647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1972ab74d562442696e2c5441012da6e, State: Initialized, Role: FOLLOWER
I20260620 14:19:54.323719 9647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972ab74d562442696e2c5441012da6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44615 } }
I20260620 14:19:54.323807 9647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:54.323834 9647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:54.323880 9647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:54.324558 9647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972ab74d562442696e2c5441012da6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44615 } }
I20260620 14:19:54.324653 9647 leader_election.cc:302] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1972ab74d562442696e2c5441012da6e; no voters:
I20260620 14:19:54.324818 9647 leader_election.cc:288] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:54.324855 9652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:54.324965 9652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [term 1 LEADER]: Becoming Leader. State: Replica: 1972ab74d562442696e2c5441012da6e, State: Running, Role: LEADER
I20260620 14:19:54.325047 9652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972ab74d562442696e2c5441012da6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44615 } }
I20260620 14:19:54.325126 9647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:54.325353 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1972ab74d562442696e2c5441012da6e. Latest consensus state: current_term: 1 leader_uuid: "1972ab74d562442696e2c5441012da6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972ab74d562442696e2c5441012da6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44615 } } }
I20260620 14:19:54.325476 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:54.325366 9653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1972ab74d562442696e2c5441012da6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1972ab74d562442696e2c5441012da6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44615 } } }
I20260620 14:19:54.325642 9653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:54.325920 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.319696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.98.62:44615)
0620 14:19:54.319914 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:54.319923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:54.319956 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:19:54.320583 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:54.320593 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:54.320679 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:54.320814 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:54.321063 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.322436 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.323623 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.324232 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.324358 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.325154 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:54.325158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:54.325158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:54.325395 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:54.325440 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:54.325454 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:54.325497 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:54.325633 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:19:54.325719 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
W20260620 14:19:54.326839 9668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:19:54.326898 9668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:19:54.326962 9657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:54.327100 9657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:54.328575 9657 catalog_manager.cc:1347] Generated new cluster ID: 7950c8b2ce7c47849ff7b7010d1351cc
I20260620 14:19:54.328630 9657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:54.350989 9657 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:54.351732 9657 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:54.357825 9657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1972ab74d562442696e2c5441012da6e: Generated new TSK 0
I20260620 14:19:54.358053 9657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:54.362965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44615
--builtin_ntp_servers=127.0.98.20:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:19:54.455857 9671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:54.456090 9671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:54.456128 9671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:54.457433 9671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:54.457528 9671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:54.457574 9671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:54.459100 9671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:54.459406 9671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:54.459781 9671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:54.465553 9677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.465565 9679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.465562 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:54.465797 9671 server_base.cc:1034] running on GCE node
I20260620 14:19:54.465996 9671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:54.466261 9671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:54.467417 9671 hybrid_clock.cc:648] HybridClock initialized: now 1781965194467407 us; error 30 us; skew 500 ppm
I20260620 14:19:54.468942 9671 webserver.cc:460] Webserver started at http://127.0.98.1:35939/ using document root <none> and password file <none>
I20260620 14:19:54.469220 9671 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:54.469266 9671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:54.469424 9671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:54.470532 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "d997c7bd06cc472c8046a3d83f2cf641"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "f39130603cd50d8c2c0c5ae2f6603dcc"
server_key_iv: "2c98f31b75345425fa188f8d22f68624"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.470893 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d997c7bd06cc472c8046a3d83f2cf641"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "f39130603cd50d8c2c0c5ae2f6603dcc"
server_key_iv: "2c98f31b75345425fa188f8d22f68624"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.472343 9671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:19:54.473140 9685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.473294 9671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:54.473382 9671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "d997c7bd06cc472c8046a3d83f2cf641"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "f39130603cd50d8c2c0c5ae2f6603dcc"
server_key_iv: "2c98f31b75345425fa188f8d22f68624"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.473455 9671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:54.486903 9671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:54.487248 9671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:54.487406 9671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:54.487700 9671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:54.488042 9671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:54.488101 9671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.488137 9671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:54.488157 9671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.494642 9671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43589
I20260620 14:19:54.494696 9798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43589 every 8 connection(s)
I20260620 14:19:54.495076 9671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:54.498405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9671
I20260620 14:19:54.498490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:54.498651 392 external_mini_cluster.cc:1428] Setting key d9bb1a4a16ff27a6062670c8dc4a17e6
I20260620 14:19:54.499830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44615
--builtin_ntp_servers=127.0.98.20:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:19:54.501556 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.495919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47919 (local address 127.0.98.62:44615)
0620 14:19:54.495985 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:54.495987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:54.496265 (+ 278us) server_negotiation.cc:407] Connection header received
0620 14:19:54.496845 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:54.496846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:54.496858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:54.496921 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:54.497648 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.498169 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.500014 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.500231 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.500311 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.501289 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:54.501290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:54.501291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:54.501295 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:54.501309 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:54.501313 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:54.501364 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:54.501475 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:19:54.501498 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:54.501856 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.495939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44615 (local address 127.0.98.1:47919)
0620 14:19:54.496128 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:54.496163 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:54.496726 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:54.497014 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:54.497017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:54.497025 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:54.497530 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.497534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.498668 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.498669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.499863 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.499865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.500326 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.500327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.500463 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.501106 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:54.501107 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:54.501151 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:54.501183 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:54.501384 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:54.501390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:54.501394 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:54.501474 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:19:54.501594 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260620 14:19:54.502472 9799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44615
I20260620 14:19:54.502555 9799 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:54.502756 9799 heartbeater.cc:511] Master 127.0.98.62:44615 requested a full tablet report, sending...
I20260620 14:19:54.503309 9611 ts_manager.cc:194] Registered new tserver with Master: d997c7bd06cc472c8046a3d83f2cf641 (127.0.98.1:43589)
I20260620 14:19:54.504338 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47919
W20260620 14:19:54.581826 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:54.582026 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:54.582046 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:54.583118 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:54.583149 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:54.583173 9802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:54.584504 9802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:54.584679 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:54.584962 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:54.590163 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.590179 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.590183 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:54.590333 9802 server_base.cc:1034] running on GCE node
I20260620 14:19:54.590566 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:54.590811 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:54.591965 9802 hybrid_clock.cc:648] HybridClock initialized: now 1781965194591955 us; error 47 us; skew 500 ppm
I20260620 14:19:54.593168 9802 webserver.cc:460] Webserver started at http://127.0.98.2:45379/ using document root <none> and password file <none>
I20260620 14:19:54.593395 9802 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:54.593463 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:54.593618 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:54.594661 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "5e64825599424502a82a6f896111e717"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "7da719a57e343dbd97dd167f616b61e6"
server_key_iv: "6978eb835e5faf34c02baa0f7a6ff2b8"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.595033 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5e64825599424502a82a6f896111e717"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "7da719a57e343dbd97dd167f616b61e6"
server_key_iv: "6978eb835e5faf34c02baa0f7a6ff2b8"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.596604 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:54.597419 9816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.597724 9802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:54.597822 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "5e64825599424502a82a6f896111e717"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "7da719a57e343dbd97dd167f616b61e6"
server_key_iv: "6978eb835e5faf34c02baa0f7a6ff2b8"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.597908 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:54.620256 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:54.620585 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:54.620741 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:54.620980 9802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:54.621311 9802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:54.621368 9802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.621440 9802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:54.621471 9802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.627485 9802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42187
I20260620 14:19:54.627529 9929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42187 every 8 connection(s)
I20260620 14:19:54.627898 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:54.633211 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.628611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38089 (local address 127.0.98.62:44615)
0620 14:19:54.628676 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:54.628678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:54.628906 (+ 228us) server_negotiation.cc:407] Connection header received
0620 14:19:54.629480 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:54.629482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:54.629495 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:54.629540 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:54.630241 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.630739 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.631754 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.631962 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.632048 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.632927 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:54.632929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:54.632929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:54.632934 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:54.632949 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:54.632953 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:54.633008 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:54.633105 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:19:54.633132 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:19:54.633443 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.628614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44615 (local address 127.0.98.2:38089)
0620 14:19:54.628782 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:54.628816 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:54.629346 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:54.629611 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:54.629620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:54.629628 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:54.630123 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.630128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.630845 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.630846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.631647 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.631649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.632055 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.632056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.632188 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.632775 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:54.632775 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:54.632804 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:54.632826 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:54.633010 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:54.633015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:54.633018 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:54.633105 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:19:54.633206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260620 14:19:54.633872 9930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44615
I20260620 14:19:54.633948 9930 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:54.634153 9930 heartbeater.cc:511] Master 127.0.98.62:44615 requested a full tablet report, sending...
I20260620 14:19:54.634480 9611 ts_manager.cc:194] Registered new tserver with Master: 5e64825599424502a82a6f896111e717 (127.0.98.2:42187)
I20260620 14:19:54.634850 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38089
I20260620 14:19:54.635622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9802
I20260620 14:19:54.635730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:54.635931 392 external_mini_cluster.cc:1428] Setting key 578d338f541e1797bdf73c554b414bcc
I20260620 14:19:54.636761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44615
--builtin_ntp_servers=127.0.98.20:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:19:54.716173 9933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:54.716362 9933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:54.716382 9933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:54.717484 9933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:54.717522 9933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:54.717547 9933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:54.718863 9933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:54.719097 9933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:54.719421 9933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:54.724213 9941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.724223 9938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.724228 9939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:54.724606 9933 server_base.cc:1034] running on GCE node
I20260620 14:19:54.724787 9933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:54.725001 9933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:54.726132 9933 hybrid_clock.cc:648] HybridClock initialized: now 1781965194726118 us; error 31 us; skew 500 ppm
I20260620 14:19:54.727686 9933 webserver.cc:460] Webserver started at http://127.0.98.3:32771/ using document root <none> and password file <none>
I20260620 14:19:54.727952 9933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:54.728003 9933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:54.728133 9933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:54.729310 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f18fd08b1754e789f239f219e3817f2"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "a2aa122b1f6ff7e250811947c94ddd52"
server_key_iv: "9351d95873a92736b5c397c639e5b1e2"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.729684 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f18fd08b1754e789f239f219e3817f2"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "a2aa122b1f6ff7e250811947c94ddd52"
server_key_iv: "9351d95873a92736b5c397c639e5b1e2"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.731105 9933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:54.732007 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.732178 9933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:19:54.732285 9933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "0f18fd08b1754e789f239f219e3817f2"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "a2aa122b1f6ff7e250811947c94ddd52"
server_key_iv: "9351d95873a92736b5c397c639e5b1e2"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.732365 9933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:54.758942 9933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:54.759284 9933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:54.759450 9933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:54.759734 9933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:54.760078 9933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:54.760133 9933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.760176 9933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:54.760202 9933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.766095 9933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43067
I20260620 14:19:54.766153 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43067 every 8 connection(s)
I20260620 14:19:54.766475 9933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:54.771476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 9933
I20260620 14:19:54.771605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:54.771800 392 external_mini_cluster.cc:1428] Setting key 888038013545ddc87aab336de367f778
I20260620 14:19:54.771801 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.767254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34183 (local address 127.0.98.62:44615)
0620 14:19:54.767307 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:54.767310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:54.767576 (+ 266us) server_negotiation.cc:407] Connection header received
0620 14:19:54.768061 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:54.768063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:54.768073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:54.768126 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:54.768888 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.769363 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.770305 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.770485 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.770546 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.771421 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:54.771422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:54.771422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:54.771426 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:54.771437 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:54.771441 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:54.771480 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:54.771689 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:19:54.771713 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:19:54.771930 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.767232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44615 (local address 127.0.98.3:34183)
0620 14:19:54.767444 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:54.767480 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:54.767975 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:54.768179 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:54.768183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:54.768191 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:54.768804 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.768810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.769456 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.769456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.770229 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:54.770231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:54.770555 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:54.770556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:54.770663 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:54.771240 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:54.771240 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:54.771296 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:54.771325 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:54.771485 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:54.771489 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:54.771493 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:54.771587 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:19:54.771700 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":103,"threads_started":1}
I20260620 14:19:54.772390 10061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44615
I20260620 14:19:54.772487 10061 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:54.772706 10061 heartbeater.cc:511] Master 127.0.98.62:44615 requested a full tablet report, sending...
I20260620 14:19:54.773046 9610 ts_manager.cc:194] Registered new tserver with Master: 0f18fd08b1754e789f239f219e3817f2 (127.0.98.3:43067)
I20260620 14:19:54.773430 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34183
I20260620 14:19:54.783777 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:19:54.785584 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:54.785090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.0.98.62:44615)
0620 14:19:54.785157 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:54.785159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:54.785220 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:19:54.785263 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:54.785263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:54.785272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:54.785322 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:54.785512 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39404: BlockingRecv error: recv got EOF from 127.0.0.1:39404 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:19:54.785609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9671
I20260620 14:19:54.790246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9802
I20260620 14:19:54.794554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9933
I20260620 14:19:54.798646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 9580
2026-06-20T14:19:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (620 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:54Z Disabled control of system clock
I20260620 14:19:54.825043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:19:54.905105 10072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:54.905311 10072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:54.905333 10072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:54.906509 10072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:54.906551 10072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:54.906564 10072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:54.906577 10072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:54.906632 10072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:54.907998 10072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:54.908252 10072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:54.908584 10072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:54.913373 10078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:54.913420 10080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:54.913561 10072 server_base.cc:1034] running on GCE node
W20260620 14:19:54.913434 10077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:54.913866 10072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:54.914149 10072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:54.915292 10072 hybrid_clock.cc:648] HybridClock initialized: now 1781965194915284 us; error 35 us; skew 500 ppm
I20260620 14:19:54.916604 10072 webserver.cc:460] Webserver started at http://127.0.98.62:43423/ using document root <none> and password file <none>
I20260620 14:19:54.916843 10072 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:54.916916 10072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:54.917068 10072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:54.918124 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "dab5181b1b174a28820b1731e9d40f31"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "401627770d336fc5b1d268d74bb7a690"
server_key_iv: "f22c92fe4f6e8d857edf241893d7fa07"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.918497 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "dab5181b1b174a28820b1731e9d40f31"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "401627770d336fc5b1d268d74bb7a690"
server_key_iv: "f22c92fe4f6e8d857edf241893d7fa07"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.919875 10072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:54.920578 10086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:54.920753 10072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:19:54.920855 10072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "dab5181b1b174a28820b1731e9d40f31"
format_stamp: "Formatted at 2026-06-20 14:19:54 on dist-test-slave-h6n5"
server_key: "401627770d336fc5b1d268d74bb7a690"
server_key_iv: "f22c92fe4f6e8d857edf241893d7fa07"
server_key_version: "encryptionkey@0"
I20260620 14:19:54.920929 10072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:54.924275 10072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:54.924540 10072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:54.924675 10072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:54.928330 10072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45897
I20260620 14:19:54.928360 10138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45897 every 8 connection(s)
I20260620 14:19:54.928686 10072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:54.928985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10072
I20260620 14:19:54.929093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:54.929294 392 external_mini_cluster.cc:1428] Setting key 6a3c0d5d271945ef9bf842fd619d8cba
I20260620 14:19:54.929518 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:54.932332 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:54.930187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.0.98.62:45897)
0620 14:19:54.930379 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:54.930386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:54.930409 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:19:54.930988 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:54.930995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:54.931036 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:54.931191 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:54.931915 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:54.931918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:54.931919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:54.931923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:54.931974 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:54.931985 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:54.932058 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:54.932129 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:19:54.932184 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260620 14:19:54.932778 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31: Bootstrap starting.
I20260620 14:19:54.933393 10139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:54.933612 10139 log.cc:824] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:54.934245 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31: No bootstrap required, opened a new log
I20260620 14:19:54.935488 10139 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab5181b1b174a28820b1731e9d40f31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260620 14:19:54.935693 10139 raft_consensus.cc:374] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:54.935721 10139 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab5181b1b174a28820b1731e9d40f31, State: Initialized, Role: FOLLOWER
I20260620 14:19:54.935838 10139 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab5181b1b174a28820b1731e9d40f31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260620 14:19:54.935925 10139 raft_consensus.cc:388] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:54.935946 10139 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:54.935976 10139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:54.936512 10139 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab5181b1b174a28820b1731e9d40f31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260620 14:19:54.936596 10139 leader_election.cc:302] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab5181b1b174a28820b1731e9d40f31; no voters:
I20260620 14:19:54.936734 10139 leader_election.cc:288] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:54.936969 10144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:54.937016 10139 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:54.937083 10144 raft_consensus.cc:686] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [term 1 LEADER]: Becoming Leader. State: Replica: dab5181b1b174a28820b1731e9d40f31, State: Running, Role: LEADER
I20260620 14:19:54.937211 10144 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab5181b1b174a28820b1731e9d40f31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260620 14:19:54.937532 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab5181b1b174a28820b1731e9d40f31. Latest consensus state: current_term: 1 leader_uuid: "dab5181b1b174a28820b1731e9d40f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab5181b1b174a28820b1731e9d40f31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260620 14:19:54.937618 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:54.937590 10146 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dab5181b1b174a28820b1731e9d40f31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab5181b1b174a28820b1731e9d40f31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260620 14:19:54.937676 10146 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31 [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:54.937873 10153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:54.938369 10153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:54.939884 10153 catalog_manager.cc:1347] Generated new cluster ID: fe4ce4d02ed5450e93e7e6e693c3bce8
I20260620 14:19:54.939939 10153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:54.946491 10153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:54.946664 10153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:54.950912 10153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dab5181b1b174a28820b1731e9d40f31: Generated new TSK 0
I20260620 14:19:54.951086 10153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:19:54.957904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--builtin_ntp_servers=127.0.98.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:19:55.048329 10163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:55.048542 10163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:55.048564 10163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:55.049734 10163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:55.049773 10163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:55.049799 10163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:55.051146 10163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:55.051393 10163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:55.051769 10163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:55.057410 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:55.057418 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:55.057480 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:55.057760 10163 server_base.cc:1034] running on GCE node
I20260620 14:19:55.057978 10163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:55.058218 10163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:55.059379 10163 hybrid_clock.cc:648] HybridClock initialized: now 1781965195059368 us; error 35 us; skew 500 ppm
I20260620 14:19:55.060787 10163 webserver.cc:460] Webserver started at http://127.0.98.1:40689/ using document root <none> and password file <none>
I20260620 14:19:55.061014 10163 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:55.061084 10163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:55.061219 10163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:55.062266 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "28ca520a0a4e49c2aac204e9b95f5b7e"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "bc7a4defd843acef1e0ada64c549a6b3"
server_key_iv: "233257e2ae5b22ff66d5398874dd2b51"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.062641 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "28ca520a0a4e49c2aac204e9b95f5b7e"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "bc7a4defd843acef1e0ada64c549a6b3"
server_key_iv: "233257e2ae5b22ff66d5398874dd2b51"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.064052 10163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:19:55.064822 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.064988 10163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:55.065073 10163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "28ca520a0a4e49c2aac204e9b95f5b7e"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "bc7a4defd843acef1e0ada64c549a6b3"
server_key_iv: "233257e2ae5b22ff66d5398874dd2b51"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.065147 10163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:55.068454 10163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:55.068724 10163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:55.068857 10163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:55.069082 10163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:55.069386 10163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:55.069423 10163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.069466 10163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:55.069492 10163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.076417 10163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37519
I20260620 14:19:55.076498 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37519 every 8 connection(s)
I20260620 14:19:55.076798 10163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:55.079883 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:55.077515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47863 (local address 127.0.98.62:45897)
0620 14:19:55.077557 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:55.077559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:55.077819 (+ 260us) server_negotiation.cc:407] Connection header received
0620 14:19:55.078321 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:55.078322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:55.078328 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:55.078386 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:55.079623 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:55.079625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:55.079625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:55.079630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:55.079644 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:55.079648 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:55.079707 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:55.079800 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:19:55.079815 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:19:55.080089 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:55.077497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.1:47863)
0620 14:19:55.077705 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:55.077737 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:55.078207 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:55.078468 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:55.078471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:55.078478 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:55.079448 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:55.079448 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:55.079476 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:55.079503 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:55.079708 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:55.079713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:55.079717 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:55.079809 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:19:55.079866 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260620 14:19:55.080571 10291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260620 14:19:55.080657 10291 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:55.080757 10291 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260620 14:19:55.081197 10103 ts_manager.cc:194] Registered new tserver with Master: 28ca520a0a4e49c2aac204e9b95f5b7e (127.0.98.1:37519)
I20260620 14:19:55.083453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10163
I20260620 14:19:55.083530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:55.083735 392 external_mini_cluster.cc:1428] Setting key 965067c5f26986c53420f04eef638c99
I20260620 14:19:55.084530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--builtin_ntp_servers=127.0.98.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:19:55.163750 10294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:55.163964 10294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:55.163986 10294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:55.165189 10294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:55.165231 10294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:55.165257 10294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:55.166620 10294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:55.166867 10294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:55.167162 10294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:55.172478 10299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:55.172457 10300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:55.172453 10302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:55.172494 10294 server_base.cc:1034] running on GCE node
I20260620 14:19:55.172892 10294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:55.173146 10294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:55.174306 10294 hybrid_clock.cc:648] HybridClock initialized: now 1781965195174288 us; error 31 us; skew 500 ppm
I20260620 14:19:55.175609 10294 webserver.cc:460] Webserver started at http://127.0.98.2:45573/ using document root <none> and password file <none>
I20260620 14:19:55.175853 10294 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:55.175932 10294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:55.176070 10294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:55.177119 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "6fda1ba10c0f4710a71b02fbd59ee9f7"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "ed5a233db3747d27ff89fd7ee0b2c242"
server_key_iv: "86014a880287d9c0ea792ce622faeb04"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.177489 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6fda1ba10c0f4710a71b02fbd59ee9f7"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "ed5a233db3747d27ff89fd7ee0b2c242"
server_key_iv: "86014a880287d9c0ea792ce622faeb04"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.178814 10294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:55.179636 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.179812 10294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:55.179934 10294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "6fda1ba10c0f4710a71b02fbd59ee9f7"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "ed5a233db3747d27ff89fd7ee0b2c242"
server_key_iv: "86014a880287d9c0ea792ce622faeb04"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.180022 10294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:55.183229 10294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:55.183501 10294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:55.183662 10294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:55.183877 10294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:55.184206 10294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:55.184260 10294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.184299 10294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:55.184326 10294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.189852 10294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46527
I20260620 14:19:55.189890 10421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46527 every 8 connection(s)
I20260620 14:19:55.190204 10294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:19:55.193131 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:55.190986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55303 (local address 127.0.98.62:45897)
0620 14:19:55.191023 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:55.191025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:55.191261 (+ 236us) server_negotiation.cc:407] Connection header received
0620 14:19:55.191769 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:55.191770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:55.191777 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:55.191841 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:55.192939 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:55.192940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:55.192940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:55.192945 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:55.192960 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:55.192964 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:55.193019 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:55.193068 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:19:55.193080 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:19:55.193341 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:55.190915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.2:55303)
0620 14:19:55.191120 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:55.191152 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:55.191670 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:55.191891 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:55.191895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:55.191902 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:55.192790 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:55.192791 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:55.192818 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:55.192845 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:55.193020 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:55.193022 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:55.193025 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:19:55.193074 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:19:55.193118 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260620 14:19:55.193835 10422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260620 14:19:55.193940 10422 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:55.194044 10422 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260620 14:19:55.194386 10103 ts_manager.cc:194] Registered new tserver with Master: 6fda1ba10c0f4710a71b02fbd59ee9f7 (127.0.98.2:46527)
I20260620 14:19:55.199007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10294
I20260620 14:19:55.199128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:55.199347 392 external_mini_cluster.cc:1428] Setting key c7700917995e570dd5a3d754ca98e868
I20260620 14:19:55.200294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--builtin_ntp_servers=127.0.98.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:19:55.281823 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:55.282110 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:55.282146 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:55.284158 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:55.284215 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:55.284266 10425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:55.286628 10425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:55.286897 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:55.287222 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:55.292007 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:55.292013 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:55.292045 10425 server_base.cc:1034] running on GCE node
W20260620 14:19:55.292013 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:55.292387 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:55.292672 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:55.293849 10425 hybrid_clock.cc:648] HybridClock initialized: now 1781965195293830 us; error 31 us; skew 500 ppm
I20260620 14:19:55.295107 10425 webserver.cc:460] Webserver started at http://127.0.98.3:41527/ using document root <none> and password file <none>
I20260620 14:19:55.295368 10425 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:55.295437 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:55.295617 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:55.296670 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "d5300ac8c18e4ebda4926345f47a9d17"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "495faf9d2dbe11b49e40b2ee31ded3b2"
server_key_iv: "8cc63d149e82978771ae48ab9b07a1ca"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.297037 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d5300ac8c18e4ebda4926345f47a9d17"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "495faf9d2dbe11b49e40b2ee31ded3b2"
server_key_iv: "8cc63d149e82978771ae48ab9b07a1ca"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.298308 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:55.298983 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.299151 10425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:55.299261 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "d5300ac8c18e4ebda4926345f47a9d17"
format_stamp: "Formatted at 2026-06-20 14:19:55 on dist-test-slave-h6n5"
server_key: "495faf9d2dbe11b49e40b2ee31ded3b2"
server_key_iv: "8cc63d149e82978771ae48ab9b07a1ca"
server_key_version: "encryptionkey@0"
I20260620 14:19:55.299343 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:55.302628 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:55.302892 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:55.303031 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:55.303237 10425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:55.303527 10425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:55.303601 10425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.303629 10425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:55.303658 10425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:55.309134 10425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44323
I20260620 14:19:55.309206 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44323 every 8 connection(s)
I20260620 14:19:55.309491 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:55.312212 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:55.310191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41377 (local address 127.0.98.62:45897)
0620 14:19:55.310250 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:55.310252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:55.310447 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:19:55.310857 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:55.310858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:55.310862 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:55.310932 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:55.312032 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:55.312034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:55.312034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:19:55.312038 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:55.312051 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:19:55.312055 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:55.312107 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:55.312152 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:19:55.312162 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:55.312390 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:55.310171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.3:41377)
0620 14:19:55.310331 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:55.310369 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:55.310781 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:55.310981 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:55.310984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:55.310991 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:55.311904 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:19:55.311904 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:55.311930 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:19:55.311955 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:55.312108 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:55.312110 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:55.312111 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:19:55.312159 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:19:55.312201 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260620 14:19:55.312825 10553 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260620 14:19:55.312903 10553 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:55.313059 10553 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260620 14:19:55.313388 10103 ts_manager.cc:194] Registered new tserver with Master: d5300ac8c18e4ebda4926345f47a9d17 (127.0.98.3:44323)
I20260620 14:19:55.314690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10425
I20260620 14:19:55.314770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:55.314973 392 external_mini_cluster.cc:1428] Setting key 637585b707943b9eb46a98c41bf4f998
I20260620 14:19:55.315812 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:19:55.317514 10142 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:19:55.317074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.0.98.62:45897)
0620 14:19:55.317143 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:55.317145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:55.317231 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:19:55.317272 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:55.317273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:55.317276 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:55.317322 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:55.317463 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59018: BlockingRecv error: recv got EOF from 127.0.0.1:59018 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:19:55.317569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10163
I20260620 14:19:55.322659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10294
I20260620 14:19:55.326648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10425
I20260620 14:19:55.330607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10072
2026-06-20T14:19:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (531 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:55 dist-test-slave-h6n5 krb5kdc[10563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:55 dist-test-slave-h6n5 krb5kdc[10563](info): set up 2 sockets
Jun 20 14:19:55 dist-test-slave-h6n5 krb5kdc[10563](info): commencing operation
krb5kdc: starting...
W20260620 14:19:56.369944 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:56 dist-test-slave-h6n5 krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:19:56.490545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:56.573117 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:56.573369 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:56.573407 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:56.574594 10576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:19:56.574666 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:56.574688 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:56.574709 10576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:19:56.574739 10576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:19:56.576229 10576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:56.576514 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:56.576756 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:56.581553 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:56.581560 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:56.581573 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:56.581895 10576 server_base.cc:1034] running on GCE node
I20260620 14:19:56.582132 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:56.582455 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:56.583616 10576 hybrid_clock.cc:648] HybridClock initialized: now 1781965196583605 us; error 48 us; skew 500 ppm
Jun 20 14:19:56 dist-test-slave-h6n5 krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:56.586231 10576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:19:56.586676 10576 webserver.cc:460] Webserver started at http://127.0.98.62:35353/ using document root <none> and password file <none>
I20260620 14:19:56.586915 10576 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:56.586982 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:56.587137 10576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:56.588295 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "0c05eb04a3114a2296a5f3fae6d0986a"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "9344098c4a797002c7bd29034e289ccd"
server_key_iv: "68ddc26a530b477b35bb2b5f2dd59cff"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.588661 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c05eb04a3114a2296a5f3fae6d0986a"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "9344098c4a797002c7bd29034e289ccd"
server_key_iv: "68ddc26a530b477b35bb2b5f2dd59cff"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.590022 10576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:19:56.590771 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:56.590931 10576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:56.591038 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "0c05eb04a3114a2296a5f3fae6d0986a"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "9344098c4a797002c7bd29034e289ccd"
server_key_iv: "68ddc26a530b477b35bb2b5f2dd59cff"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.591113 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:56.604763 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:56.607125 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:56.607285 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:56.611102 10576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42485
I20260620 14:19:56.611160 10643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42485 every 8 connection(s)
I20260620 14:19:56.611516 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:19:56.612705 10644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:56.614943 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a: Bootstrap starting.
I20260620 14:19:56.615407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10576
I20260620 14:19:56.615517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:19:56.615547 10644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:56.615746 392 external_mini_cluster.cc:1428] Setting key b96e23a660535a28ed9703296402b6e7
I20260620 14:19:56.615862 10644 log.cc:824] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:56.616621 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a: No bootstrap required, opened a new log
I20260620 14:19:56.618533 10644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42485 } }
I20260620 14:19:56.618662 10644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:56.618685 10644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c05eb04a3114a2296a5f3fae6d0986a, State: Initialized, Role: FOLLOWER
I20260620 14:19:56.618817 10644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42485 } }
I20260620 14:19:56.618903 10644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:19:56.618961 10644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:19:56.619009 10644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:56.619601 10644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42485 } }
I20260620 14:19:56.619724 10644 leader_election.cc:302] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c05eb04a3114a2296a5f3fae6d0986a; no voters:
I20260620 14:19:56.619899 10644 leader_election.cc:288] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:19:56.619926 10649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:56.620060 10649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [term 1 LEADER]: Becoming Leader. State: Replica: 0c05eb04a3114a2296a5f3fae6d0986a, State: Running, Role: LEADER
I20260620 14:19:56.620144 10649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42485 } }
I20260620 14:19:56.620150 10644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:19:56.620496 10649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c05eb04a3114a2296a5f3fae6d0986a. Latest consensus state: current_term: 1 leader_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42485 } } }
I20260620 14:19:56.620564 10649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [sys.catalog]: This master's current role is: LEADER
I20260620 14:19:56.620764 10654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:19:56.621232 10654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:19:56.622400 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c05eb04a3114a2296a5f3fae6d0986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42485 } } }
I20260620 14:19:56.622471 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a [sys.catalog]: This master's current role is: LEADER
Jun 20 14:19:56 dist-test-slave-h6n5 krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:56.626653 10654 catalog_manager.cc:1347] Generated new cluster ID: 2051aa6c98884830a7826a449dd39b21
I20260620 14:19:56.626720 10654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:19:56.626741 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:56.616300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.98.62:42485)
0620 14:19:56.616482 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:56.616492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:56.616524 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:19:56.617007 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:56.617013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:56.617060 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:56.617206 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:56.617802 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.618684 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.619581 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.620212 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.620349 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:56.623287 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:56.623290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:56.623291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:56.623302 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:56.625194 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:56.625546 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:56.625549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:56.625549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:56.625661 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:56.625845 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:56.625846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:56.625846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:56.626157 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:56.626319 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:56.626394 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:19:56.626529 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260620 14:19:56.642861 10654 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:19:56.643457 10654 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:19:56.649842 10654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a: Generated new TSK 0
I20260620 14:19:56.650080 10654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:19:56.674602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42485
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:56.775579 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:56.775799 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:56.775827 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:56.776945 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:56.776983 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:56.777009 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:19:56.778394 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:56.778584 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:56.778749 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:56.783772 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:56.783792 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:56.783779 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:56.783998 10670 server_base.cc:1034] running on GCE node
I20260620 14:19:56.784273 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:56.784505 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:56.785669 10670 hybrid_clock.cc:648] HybridClock initialized: now 1781965196785645 us; error 44 us; skew 500 ppm
Jun 20 14:19:56 dist-test-slave-h6n5 krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:56.788220 10670 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:19:56.788651 10670 webserver.cc:460] Webserver started at http://127.0.98.1:35075/ using document root <none> and password file <none>
I20260620 14:19:56.788875 10670 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:56.788924 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:56.789038 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:56.790088 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "537defeec3d94040a326435a0146a532"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "4f7a66ecae0a08449c20878aac50cad6"
server_key_iv: "b28d2da9bc58425fcb7ee9d37be3f4d8"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.790441 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "537defeec3d94040a326435a0146a532"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "4f7a66ecae0a08449c20878aac50cad6"
server_key_iv: "b28d2da9bc58425fcb7ee9d37be3f4d8"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.791795 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:19:56.792531 10685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:56.792721 10670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:56.792817 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "537defeec3d94040a326435a0146a532"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "4f7a66ecae0a08449c20878aac50cad6"
server_key_iv: "b28d2da9bc58425fcb7ee9d37be3f4d8"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.792899 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:56.832144 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:56.835162 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:56.835340 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:56.835640 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:56.836048 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:56.836108 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:56.836158 10670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:56.836194 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:56.844545 10670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44217
I20260620 14:19:56.844585 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44217 every 8 connection(s)
I20260620 14:19:56.844918 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:19:56.850313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10670
I20260620 14:19:56.850440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:19:56.850644 392 external_mini_cluster.cc:1428] Setting key 65504cc68420226eb60aada0867ae0fc
Jun 20 14:19:56 dist-test-slave-h6n5 krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:56.854419 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:56.845662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57457 (local address 127.0.98.62:42485)
0620 14:19:56.845730 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:56.845732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:56.845947 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:19:56.846440 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:56.846441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:56.846451 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:56.846503 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:56.847409 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.847961 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.848938 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.849123 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.849172 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:56.851408 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:56.851409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:56.851410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:56.851418 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:56.853041 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:56.853513 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:56.853514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:56.853515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:56.853565 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:56.853831 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:56.853831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:56.853832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:56.853926 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:56.853974 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:56.854290 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 14:19:56.854354 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:19:56.854717 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:56.845637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42485 (local address 127.0.98.1:57457)
0620 14:19:56.845823 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:56.845864 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:56.846328 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:56.846557 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:56.846560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:56.847134 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:56.847327 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:56.847331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.848046 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.848047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:56.848855 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:56.848857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.849190 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.849191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:56.849287 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:56.849755 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:56.849762 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:56.851307 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:56.853119 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:56.853126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:56.853128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:56.853443 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:56.853626 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:56.853627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:56.853627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:56.853769 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:56.853985 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:56.853991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:56.854160 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 14:19:56.854273 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:19:56.854444 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260620 14:19:56.855299 10799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42485
I20260620 14:19:56.855414 10799 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:56.855626 10799 heartbeater.cc:511] Master 127.0.98.62:42485 requested a full tablet report, sending...
I20260620 14:19:56.856242 10608 ts_manager.cc:194] Registered new tserver with Master: 537defeec3d94040a326435a0146a532 (127.0.98.1:44217)
I20260620 14:19:56.857228 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57457
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:19:56.874266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42485
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:56.954643 10804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:56.954851 10804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:56.954875 10804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:56.956066 10804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:56.956107 10804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:56.956135 10804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:19:56.957571 10804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:56.957809 10804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:56.958061 10804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:56.962996 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:56.963024 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:56.962996 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:56.963258 10804 server_base.cc:1034] running on GCE node
I20260620 14:19:56.963486 10804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:56.963763 10804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:56.964924 10804 hybrid_clock.cc:648] HybridClock initialized: now 1781965196964910 us; error 26 us; skew 500 ppm
Jun 20 14:19:56 dist-test-slave-h6n5 krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:56.967419 10804 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:19:56.967892 10804 webserver.cc:460] Webserver started at http://127.0.98.2:33221/ using document root <none> and password file <none>
I20260620 14:19:56.968120 10804 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:56.968182 10804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:56.968330 10804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:56.969391 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "05c28b0af1bc4a77841cac9f3902d590"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "eca11ac019e9bb37d80f27aa79cfd2cc"
server_key_iv: "b0ed4ca830ac8623f8a33caac5a7a230"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.969759 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05c28b0af1bc4a77841cac9f3902d590"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "eca11ac019e9bb37d80f27aa79cfd2cc"
server_key_iv: "b0ed4ca830ac8623f8a33caac5a7a230"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.971060 10804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:19:56.971846 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:56.972038 10804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:56.972139 10804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "05c28b0af1bc4a77841cac9f3902d590"
format_stamp: "Formatted at 2026-06-20 14:19:56 on dist-test-slave-h6n5"
server_key: "eca11ac019e9bb37d80f27aa79cfd2cc"
server_key_iv: "b0ed4ca830ac8623f8a33caac5a7a230"
server_key_version: "encryptionkey@0"
I20260620 14:19:56.972259 10804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:56.983901 10804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:56.985774 10804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:56.985929 10804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:56.986166 10804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:56.986474 10804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:56.986524 10804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:56.986567 10804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:56.986595 10804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:56.992318 10804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36411
I20260620 14:19:56.992357 10932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36411 every 8 connection(s)
I20260620 14:19:56.992667 10804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:19:56 dist-test-slave-h6n5 krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:56.999045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10804
I20260620 14:19:56.999163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:19:56.999400 392 external_mini_cluster.cc:1428] Setting key c68b30ea33c3911df2250d8053e5f8e6
I20260620 14:19:57.002316 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:56.993304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44721 (local address 127.0.98.62:42485)
0620 14:19:56.993375 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:56.993378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:56.993597 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:19:56.994134 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:56.994135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:56.994147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:56.994204 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:56.995199 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.995809 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.996787 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.997044 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.997124 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:56.999233 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:56.999235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:56.999235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:56.999243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:57.000819 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:57.001308 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:57.001309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:57.001309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:57.001377 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:57.001666 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:57.001667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:57.001667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:57.001809 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:57.001880 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.002109 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.002196 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:19:57.002336 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:56.993302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42485 (local address 127.0.98.2:44721)
0620 14:19:56.993474 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:56.993508 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:56.994027 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:56.994280 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:56.994283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:56.994914 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:56.995096 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:56.995101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.995887 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.995888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:56.996689 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:56.996691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:56.997132 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:56.997134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:56.997256 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:56.997774 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:56.997783 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:56.999118 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:57.000900 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:57.000906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:57.000909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:57.001228 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:57.001505 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:57.001506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:57.001506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:57.001603 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:57.001874 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:57.001877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:57.001987 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:19:57.002067 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:19:57.002181 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260620 14:19:57.002866 10933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42485
I20260620 14:19:57.002975 10933 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:57.003160 10933 heartbeater.cc:511] Master 127.0.98.62:42485 requested a full tablet report, sending...
I20260620 14:19:57.003522 10608 ts_manager.cc:194] Registered new tserver with Master: 05c28b0af1bc4a77841cac9f3902d590 (127.0.98.2:36411)
I20260620 14:19:57.003916 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44721
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:19:57.023993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42485
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:19:57.105058 10938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:19:57.105280 10938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:19:57.105305 10938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:19:57.106479 10938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:19:57.106523 10938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:19:57.106551 10938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:19:57.108078 10938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:19:57.108359 10938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:19:57.108599 10938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:19:57.113503 10944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:19:57.113511 10943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:57.113581 10938 server_base.cc:1034] running on GCE node
W20260620 14:19:57.113677 10946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:19:57.113919 10938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:19:57.114157 10938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:19:57.115314 10938 hybrid_clock.cc:648] HybridClock initialized: now 1781965197115270 us; error 65 us; skew 500 ppm
Jun 20 14:19:57 dist-test-slave-h6n5 krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:19:57.117969 10938 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:19:57.118458 10938 webserver.cc:460] Webserver started at http://127.0.98.3:35013/ using document root <none> and password file <none>
I20260620 14:19:57.118641 10938 fs_manager.cc:359] Metadata directory not provided
I20260620 14:19:57.118681 10938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:19:57.118769 10938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:19:57.119865 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "a1e3d2c2e74849fab121a95b96ef3d03"
format_stamp: "Formatted at 2026-06-20 14:19:57 on dist-test-slave-h6n5"
server_key: "064ba21d3306f2349d0299f426cdf510"
server_key_iv: "a41b599b53c105ce105d294989245bbf"
server_key_version: "encryptionkey@0"
I20260620 14:19:57.120220 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a1e3d2c2e74849fab121a95b96ef3d03"
format_stamp: "Formatted at 2026-06-20 14:19:57 on dist-test-slave-h6n5"
server_key: "064ba21d3306f2349d0299f426cdf510"
server_key_iv: "a41b599b53c105ce105d294989245bbf"
server_key_version: "encryptionkey@0"
I20260620 14:19:57.121654 10938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:19:57.122498 10953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:57.122679 10938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:19:57.122781 10938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "a1e3d2c2e74849fab121a95b96ef3d03"
format_stamp: "Formatted at 2026-06-20 14:19:57 on dist-test-slave-h6n5"
server_key: "064ba21d3306f2349d0299f426cdf510"
server_key_iv: "a41b599b53c105ce105d294989245bbf"
server_key_version: "encryptionkey@0"
I20260620 14:19:57.122846 10938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:19:57.134881 10938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:19:57.137181 10938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:19:57.137292 10938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:19:57.137470 10938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:19:57.137723 10938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:19:57.137758 10938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:57.137780 10938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:19:57.137792 10938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:19:57.143673 10938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39279
I20260620 14:19:57.143721 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39279 every 8 connection(s)
I20260620 14:19:57.144037 10938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:19:57.148869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 10938
I20260620 14:19:57.148978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:19:57.149180 392 external_mini_cluster.cc:1428] Setting key 2c618837192cd81eb728b3de0ce7df3a
Jun 20 14:19:57 dist-test-slave-h6n5 krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:19:57.153119 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.144743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57333 (local address 127.0.98.62:42485)
0620 14:19:57.144806 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.144808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.144996 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:19:57.145452 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.145454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.145464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.145513 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:57.146423 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.146913 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.147922 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.148103 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.148159 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.150561 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:57.150562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:57.150562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:57.150569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:57.152025 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:57.152368 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:57.152369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:57.152369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:57.152419 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:57.152620 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:57.152620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:57.152620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:57.152715 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:57.152763 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.152992 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.153056 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:19:57.153285 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.144736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42485 (local address 127.0.98.3:57333)
0620 14:19:57.144876 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:57.144911 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:57.145371 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:57.145578 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:57.145581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:57.146142 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:19:57.146339 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.146344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.146998 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.146999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.147842 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.147845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.148194 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.148195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.148302 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.148818 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:19:57.148830 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:19:57.150483 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:19:57.152097 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:57.152102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:57.152103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:57.152311 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:57.152474 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:19:57.152475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:19:57.152475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:19:57.152567 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:19:57.152764 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:19:57.152767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:19:57.152877 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:19:57.152969 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:19:57.153086 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 14:19:57.153713 11067 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42485
I20260620 14:19:57.153815 11067 heartbeater.cc:464] Registering TS with master...
I20260620 14:19:57.153988 11067 heartbeater.cc:511] Master 127.0.98.62:42485 requested a full tablet report, sending...
I20260620 14:19:57.154333 10608 ts_manager.cc:194] Registered new tserver with Master: a1e3d2c2e74849fab121a95b96ef3d03 (127.0.98.3:39279)
I20260620 14:19:57.154711 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57333
I20260620 14:19:57.161051 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:19:57.167642 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.162273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.0.98.62:42485)
0620 14:19:57.162308 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.162310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.162442 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:19:57.162470 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.162471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.162480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.162530 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:19:57.163013 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.163474 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.164214 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.164385 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.164438 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.165544 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:19:57.165544 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:19:57.165545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:19:57.165550 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:19:57.166688 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:57.166988 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:57.166989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:57.166989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:57.167035 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:19:57.167185 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:19:57.167186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:19:57.167186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:19:57.167308 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:19:57.167381 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.167464 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.167516 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:19:57.169243 10608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:19:57.170755 10608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:19:57.177361 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.173104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.0.98.2:36411)
0620 14:19:57.173512 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.173517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.173528 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:19:57.173570 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.173572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.173588 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.173669 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:57.173941 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.175107 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.176471 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.176953 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.177049 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.177110 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.177236 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.177275 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":63,"threads_started":1}
I20260620 14:19:57.177382 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.172985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36411 (local address 127.0.0.1:40350)
0620 14:19:57.173304 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:57.173316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:57.173372 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:57.173819 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:57.173820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:57.173821 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:57.173859 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.173861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.175204 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.175205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.176396 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.176398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.177051 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.177053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.177129 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.177142 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:19:57.177209 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:19:57.177269 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":58,"thread_start_us":86,"threads_started":1}
I20260620 14:19:57.177563 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.172956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39279 (local address 127.0.0.1:41496)
0620 14:19:57.173270 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:57.173307 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:57.173368 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:57.173669 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:57.173672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:57.173675 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:57.173739 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.173741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.174901 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.174902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.176230 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.176232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.177138 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.177138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.177210 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.177214 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:19:57.177245 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:19:57.177301 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":31,"thread_start_us":65,"threads_started":1}
I20260620 14:19:57.177572 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.173297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.98.1:44217)
0620 14:19:57.173484 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.173490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.173553 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:19:57.173622 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.173624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.173638 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.173830 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:57.174104 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.175168 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.176350 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.177035 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.177108 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.177162 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.177432 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.177482 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260620 14:19:57.177699 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.173180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44217 (local address 127.0.0.1:37658)
0620 14:19:57.173506 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:57.173514 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:57.173559 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:57.173826 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:57.173827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:57.173829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:57.173875 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.173877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.175281 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.175282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.176245 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.176248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.177190 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.177190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.177242 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.177248 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:19:57.177293 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:19:57.177322 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":60,"threads_started":1}
I20260620 14:19:57.178066 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.173105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.98.3:39279)
0620 14:19:57.173521 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.173525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.173534 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:19:57.173570 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.173572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.173588 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.173669 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:57.173946 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.174745 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.176317 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.177036 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.177555 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.177743 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.177770 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.177824 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":115,"threads_started":1}
I20260620 14:19:57.178300 10867 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1d39e0abf74d609f6b66f01a29f5d8 (DEFAULT_TABLE table=test-table [id=3efbba55aac7487a98fdf104c6dd207f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:57.178357 10733 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1d39e0abf74d609f6b66f01a29f5d8 (DEFAULT_TABLE table=test-table [id=3efbba55aac7487a98fdf104c6dd207f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:57.178570 10867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1d39e0abf74d609f6b66f01a29f5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:57.178654 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1d39e0abf74d609f6b66f01a29f5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:57.181210 11084 tablet_bootstrap.cc:492] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Bootstrap starting.
I20260620 14:19:57.181555 11085 tablet_bootstrap.cc:492] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532: Bootstrap starting.
I20260620 14:19:57.181799 11001 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1d39e0abf74d609f6b66f01a29f5d8 (DEFAULT_TABLE table=test-table [id=3efbba55aac7487a98fdf104c6dd207f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:19:57.181937 11084 tablet_bootstrap.cc:654] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:57.182152 11085 tablet_bootstrap.cc:654] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:57.182139 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1d39e0abf74d609f6b66f01a29f5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:19:57.182498 11084 log.cc:824] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:57.182664 11085 log.cc:824] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:57.183311 11084 tablet_bootstrap.cc:492] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: No bootstrap required, opened a new log
I20260620 14:19:57.183386 11084 ts_tablet_manager.cc:1397] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:57.183408 11085 tablet_bootstrap.cc:492] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532: No bootstrap required, opened a new log
I20260620 14:19:57.183458 11085 ts_tablet_manager.cc:1397] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:57.185043 11088 tablet_bootstrap.cc:492] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Bootstrap starting.
I20260620 14:19:57.185397 11085 raft_consensus.cc:348] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.185390 11084 raft_consensus.cc:348] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.185578 11085 raft_consensus.cc:374] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:57.185578 11084 raft_consensus.cc:374] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:57.185616 11085 raft_consensus.cc:729] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537defeec3d94040a326435a0146a532, State: Initialized, Role: FOLLOWER
I20260620 14:19:57.185616 11084 raft_consensus.cc:729] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c28b0af1bc4a77841cac9f3902d590, State: Initialized, Role: FOLLOWER
I20260620 14:19:57.185693 11088 tablet_bootstrap.cc:654] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Neither blocks nor log segments found. Creating new log.
I20260620 14:19:57.185732 11084 consensus_queue.cc:260] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.185757 11085 consensus_queue.cc:260] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.186012 11085 ts_tablet_manager.cc:1428] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:19:57.186012 11084 ts_tablet_manager.cc:1428] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:19:57.186134 10799 heartbeater.cc:503] Master 127.0.98.62:42485 was elected leader, sending a full tablet report...
I20260620 14:19:57.186267 11088 log.cc:824] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Log is configured to *not* fsync() on all Append() calls
I20260620 14:19:57.186427 10933 heartbeater.cc:503] Master 127.0.98.62:42485 was elected leader, sending a full tablet report...
I20260620 14:19:57.187392 11088 tablet_bootstrap.cc:492] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: No bootstrap required, opened a new log
I20260620 14:19:57.187471 11088 ts_tablet_manager.cc:1397] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:19:57.188992 11088 raft_consensus.cc:348] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.189154 11088 raft_consensus.cc:374] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:19:57.189200 11088 raft_consensus.cc:729] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e3d2c2e74849fab121a95b96ef3d03, State: Initialized, Role: FOLLOWER
I20260620 14:19:57.189312 11088 consensus_queue.cc:260] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.189553 11088 ts_tablet_manager.cc:1428] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:19:57.189585 11067 heartbeater.cc:503] Master 127.0.98.62:42485 was elected leader, sending a full tablet report...
W20260620 14:19:57.243286 10934 tablet.cc:2369] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:19:57.345667 10800 tablet.cc:2369] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:19:57.394557 11068 tablet.cc:2369] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:19:57.410138 11090 raft_consensus.cc:482] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:19:57.410257 11090 raft_consensus.cc:504] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.410677 11090 leader_election.cc:288] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1e3d2c2e74849fab121a95b96ef3d03 (127.0.98.3:39279), 537defeec3d94040a326435a0146a532 (127.0.98.1:44217)
I20260620 14:19:57.414685 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.410972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60653 (local address 127.0.98.1:44217)
0620 14:19:57.411035 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.411037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.411174 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:19:57.411225 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.411226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.411241 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.411313 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:57.411472 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.412242 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.413750 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.414312 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.414388 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.414399 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.414571 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.414612 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:19:57.414685 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.410986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33091 (local address 127.0.98.3:39279)
0620 14:19:57.411029 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.411031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.411043 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:19:57.411108 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.411109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.411118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.411209 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:19:57.411463 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.412237 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.413749 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.414312 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.414388 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.414426 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.414571 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.414611 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:19:57.414705 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.410914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39279 (local address 127.0.98.2:33091)
0620 14:19:57.410991 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:57.411004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:57.411046 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:57.411221 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:57.411222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:57.411224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:57.411345 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.411347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.412322 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.412323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.413655 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.413658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.414396 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.414397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.414495 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.414497 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:19:57.414553 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:19:57.414610 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":100}
I20260620 14:19:57.414716 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.410914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44217 (local address 127.0.98.2:60653)
0620 14:19:57.411095 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:57.411104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:19:57.411157 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:19:57.411339 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:19:57.411340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:19:57.411341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:19:57.411402 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.411403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.412322 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.412323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.413655 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:19:57.413658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.414396 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.414397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:19:57.414490 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.414495 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:19:57.414553 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:19:57.414610 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":93,"thread_start_us":107,"threads_started":1}
I20260620 14:19:57.415131 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1d39e0abf74d609f6b66f01a29f5d8" candidate_uuid: "05c28b0af1bc4a77841cac9f3902d590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "537defeec3d94040a326435a0146a532" is_pre_election: true
I20260620 14:19:57.415112 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1d39e0abf74d609f6b66f01a29f5d8" candidate_uuid: "05c28b0af1bc4a77841cac9f3902d590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" is_pre_election: true
I20260620 14:19:57.415277 10753 raft_consensus.cc:2454] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05c28b0af1bc4a77841cac9f3902d590 in term 0.
I20260620 14:19:57.415277 11021 raft_consensus.cc:2454] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05c28b0af1bc4a77841cac9f3902d590 in term 0.
I20260620 14:19:57.415514 10820 leader_election.cc:302] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05c28b0af1bc4a77841cac9f3902d590, 537defeec3d94040a326435a0146a532; no voters:
I20260620 14:19:57.415702 11090 raft_consensus.cc:2781] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:19:57.415764 11090 raft_consensus.cc:482] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:19:57.415787 11090 raft_consensus.cc:3037] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:57.416496 11090 raft_consensus.cc:504] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.416639 11090 leader_election.cc:288] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [CANDIDATE]: Term 1 election: Requested vote from peers a1e3d2c2e74849fab121a95b96ef3d03 (127.0.98.3:39279), 537defeec3d94040a326435a0146a532 (127.0.98.1:44217)
I20260620 14:19:57.416798 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1d39e0abf74d609f6b66f01a29f5d8" candidate_uuid: "05c28b0af1bc4a77841cac9f3902d590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "537defeec3d94040a326435a0146a532"
I20260620 14:19:57.416847 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1d39e0abf74d609f6b66f01a29f5d8" candidate_uuid: "05c28b0af1bc4a77841cac9f3902d590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e3d2c2e74849fab121a95b96ef3d03"
I20260620 14:19:57.416908 11021 raft_consensus.cc:3037] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:57.416899 10753 raft_consensus.cc:3037] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:19:57.417784 10753 raft_consensus.cc:2454] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05c28b0af1bc4a77841cac9f3902d590 in term 1.
I20260620 14:19:57.417784 11021 raft_consensus.cc:2454] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05c28b0af1bc4a77841cac9f3902d590 in term 1.
I20260620 14:19:57.417960 10820 leader_election.cc:302] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05c28b0af1bc4a77841cac9f3902d590, 537defeec3d94040a326435a0146a532; no voters:
I20260620 14:19:57.418066 11090 raft_consensus.cc:2781] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:19:57.418185 11090 raft_consensus.cc:686] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 1 LEADER]: Becoming Leader. State: Replica: 05c28b0af1bc4a77841cac9f3902d590, State: Running, Role: LEADER
I20260620 14:19:57.418388 11090 consensus_queue.cc:237] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } }
I20260620 14:19:57.419181 10608 catalog_manager.cc:5495] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05c28b0af1bc4a77841cac9f3902d590 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "05c28b0af1bc4a77841cac9f3902d590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05c28b0af1bc4a77841cac9f3902d590" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 } health_report { overall_health: UNKNOWN } } }
I20260620 14:19:57.458473 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.456308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.0.98.2:36411)
0620 14:19:57.456348 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:19:57.456350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:19:57.456358 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:19:57.456428 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:19:57.456429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:19:57.456449 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:19:57.456507 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:19:57.456611 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.457068 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.457771 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:19:57.457975 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:19:57.458032 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:19:57.458151 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:19:57.458248 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:19:57.458317 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:19:57.458333 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 14:19:57.458348 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:19:57.460387 11021 raft_consensus.cc:1261] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 1 FOLLOWER]: Refusing update from remote peer 05c28b0af1bc4a77841cac9f3902d590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:57.460387 10753 raft_consensus.cc:1261] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 537defeec3d94040a326435a0146a532 [term 1 FOLLOWER]: Refusing update from remote peer 05c28b0af1bc4a77841cac9f3902d590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:19:57.460785 11094 consensus_queue.cc:1035] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "537defeec3d94040a326435a0146a532" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:57.460922 11090 consensus_queue.cc:1035] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e3d2c2e74849fab121a95b96ef3d03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:19:57.464326 11097 mvcc.cc:205] Tried to move back new op lower bound from 7298929448794783744 to 7298929448626237440. Current Snapshot: MvccSnapshot[applied={T|T < 7298929448794783744 or (T in {7298929448794783744})}]
I20260620 14:19:57.465291 10607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:57.465385 10607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:19:57.466102 10607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c05eb04a3114a2296a5f3fae6d0986a: Sending DeleteTablet for 3 replicas of tablet 5f1d39e0abf74d609f6b66f01a29f5d8
I20260620 14:19:57.466414 11001 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1d39e0abf74d609f6b66f01a29f5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41496
I20260620 14:19:57.466495 10867 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1d39e0abf74d609f6b66f01a29f5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40350
I20260620 14:19:57.466501 10733 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1d39e0abf74d609f6b66f01a29f5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:19:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37658
I20260620 14:19:57.466761 11107 tablet_replica.cc:330] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: stopping tablet replica
I20260620 14:19:57.466867 11107 raft_consensus.cc:2229] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:19:57.466864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10670
I20260620 14:19:57.468636 11108 tablet_replica.cc:330] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: stopping tablet replica
I20260620 14:19:57.468735 11108 raft_consensus.cc:2229] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:19:57.468820 11108 raft_consensus.cc:2258] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:57.469007 11107 raft_consensus.cc:2258] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:19:57.469264 11108 ts_tablet_manager.cc:1905] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:57.469427 11107 ts_tablet_manager.cc:1905] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:19:57.471030 11108 ts_tablet_manager.cc:1918] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:57.471107 11108 log.cc:1196] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/5f1d39e0abf74d609f6b66f01a29f5d8
I20260620 14:19:57.471072 11107 ts_tablet_manager.cc:1918] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:19:57.471295 11107 log.cc:1196] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/5f1d39e0abf74d609f6b66f01a29f5d8
I20260620 14:19:57.471371 11108 ts_tablet_manager.cc:1939] T 5f1d39e0abf74d609f6b66f01a29f5d8 P 05c28b0af1bc4a77841cac9f3902d590: Deleting consensus metadata
I20260620 14:19:57.471706 11107 ts_tablet_manager.cc:1939] T 5f1d39e0abf74d609f6b66f01a29f5d8 P a1e3d2c2e74849fab121a95b96ef3d03: Deleting consensus metadata
I20260620 14:19:57.471794 10592 catalog_manager.cc:4841] TS 05c28b0af1bc4a77841cac9f3902d590 (127.0.98.2:36411): tablet 5f1d39e0abf74d609f6b66f01a29f5d8 (table test-table [id=3efbba55aac7487a98fdf104c6dd207f]) successfully deleted
I20260620 14:19:57.472200 10593 catalog_manager.cc:4841] TS a1e3d2c2e74849fab121a95b96ef3d03 (127.0.98.3:39279): tablet 5f1d39e0abf74d609f6b66f01a29f5d8 (table test-table [id=3efbba55aac7487a98fdf104c6dd207f]) successfully deleted
W20260620 14:19:57.474213 10592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44217 (error 108)
I20260620 14:19:57.474472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10804
I20260620 14:19:57.475162 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:19:57.474580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44217 (local address 127.0.0.1:37660)
0620 14:19:57.475084 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 14:19:57.475123 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484}
W20260620 14:19:57.475464 10592 catalog_manager.cc:4584] TS 537defeec3d94040a326435a0146a532 (127.0.98.1:44217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f1d39e0abf74d609f6b66f01a29f5d8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44217: connect: Connection refused (error 111)
I20260620 14:19:57.479944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10938
I20260620 14:19:57.484381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 10576
2026-06-20T14:19:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2154 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:19:57 dist-test-slave-h6n5 krb5kdc[11113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:19:57 dist-test-slave-h6n5 krb5kdc[11113](info): set up 2 sockets
Jun 20 14:19:57 dist-test-slave-h6n5 krb5kdc[11113](info): commencing operation
krb5kdc: starting...
W20260620 14:19:58.520574 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:19:58 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:19:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:19:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:19:58.678382 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTTnugX/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:20:01.134 UTC [11143] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:20:01.134 UTC [11143] LOG: listening on IPv4 address "127.0.98.20", port 38447
2026-06-20 14:20:01.138 UTC [11143] LOG: listening on Unix socket "/tmp/.s.PGSQL.38447"
2026-06-20 14:20:01.160 UTC [11146] LOG: database system was shut down at 2026-06-20 14:19:59 UTC
2026-06-20 14:20:01.163 UTC [11143] LOG: database system is ready to accept connections
I20260620 14:20:02.117067 392 mini_postgres.cc:96] Postgres bound to 38447
2026-06-20 14:20:02.123 UTC [11154] FATAL: database "slave" does not exist
127.0.98.20:38447 - accepting connections
I20260620 14:20:02.123838 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:20:02.137869 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:20:02.627251 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:20:02.627329 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-admin
2026-06-20 14:20:02,920 [I] DB FLAVOR :POSTGRES
2026-06-20 14:20:02,920 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:20:02,920 [I] Checking connection..
2026-06-20 14:20:02,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:20:03,164 [I] Checking connection passed.
2026-06-20 14:20:03,164 [I] --------- Verifying version history table ---------
2026-06-20 14:20:03,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:20:03,404 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:20:03,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:20:03,668 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:20:03,668 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:20:03,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:20:03.891 UTC [11218] WARNING: there is no transaction in progress
2026-06-20 14:20:03,897 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:20:03,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:20:04,144 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:20:04,145 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:20:04,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:20:04,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:20:04,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h6n5.c.gcp-upstream.internal','N') ;"
2026-06-20 14:20:04,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:20:05,078 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:20:05,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:20:05,324 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:20:05,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:20:05,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:20:05,818 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:20:05,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:20:06.788 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.805 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.823 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.839 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.860 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.882 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.943 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.950 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.965 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.975 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:06.986 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:07.001 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:07.010 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:07.017 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:07.024 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:07.031 UTC [11326] WARNING: there is no transaction in progress
2026-06-20 14:20:07,670 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:20:07,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:20:07,921 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:20:07,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:20:08,167 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:20:08,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:20:08,396 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:20:08,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:20:08,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:20:08,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h6n5.c.gcp-upstream.internal';"
2026-06-20 14:20:09,127 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:20:09,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:20:09,371 [I] DB_PATCHES have already been applied
I20260620 14:20:09.376566 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTTnugX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:20:09.376631 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:20:09.379635 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51681
Jun 20, 2026 2:20:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:20:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:20:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:20:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:20:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:20:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:20:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:20:10.483264 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.359s user 0.000s sys 0.008s
Jun 20, 2026 2:20:30 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:20:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:20:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:20:30 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:20:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:20:31.611030 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:20:31 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965231, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:20:31.703289 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:20:31.713533 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:20:32.560433 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:20:32.560528 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms
I20260620 14:20:32.560546 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:38447
2026-06-20 14:20:32,735 [I] DB FLAVOR :POSTGRES
2026-06-20 14:20:32,736 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:20:32,736 [I] Checking connection
2026-06-20 14:20:32,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:20:33,002 [I] connection success
2026-06-20 14:20:33,002 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:20:33,003 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:20:33,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:20:33,255 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:20:33,256 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:20:33,256 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:20:33,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:20:33,549 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:20:33.977666 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:20:34.177778 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:20:34.221654 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:20:34 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965231, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:20:34.621887 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:20:34.621999 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:20:34.622020 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:20:34.625252 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11506
I20260620 14:20:34.625387 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54653
Jun 20, 2026 2:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:20:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:20:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:20:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:20:36.931496 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.228s user 0.000s sys 0.158s
I20260620 14:20:36.931630 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:20:36.931710 392 mini_ranger_kms.cc:401] 127.0.98.20:54653/kms/v1/keys
Jun 20 14:20:37 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965237, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:20:37 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965237, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:20:38 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965237, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:20:40 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:20:40.234449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40793
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40793
--ranger_config_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54653/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:40.337819 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:40.338021 11557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:40.338044 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:40.339182 11557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:20:40.339222 11557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:40.339236 11557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:40.339249 11557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:20:40.339262 11557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:20:40.340854 11557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54653/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40793
--ranger_config_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:20:40.341074 11557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:40.341296 11557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:20:40.347508 11557 server_base.cc:1034] running on GCE node
W20260620 14:20:40.347463 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:40.347702 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:40.347443 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:40.347978 11557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:40.348265 11557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:40.349455 11557 hybrid_clock.cc:648] HybridClock initialized: now 1781965240349439 us; error 34 us; skew 500 ppm
Jun 20 14:20:40 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:40.352160 11557 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:20:40.352694 11557 webserver.cc:460] Webserver started at http://127.0.98.62:43433/ using document root <none> and password file <none>
I20260620 14:20:40.352945 11557 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:40.352998 11557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:40.353117 11557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:20:40 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:20:40.406051 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "04b1d19afced46e9b2a5dc77e913c059"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-h6n5"
server_key: "b3556027389f390400853cb290af8538"
server_key_iv: "42bc613e42f922722ae3dc118873e827"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:40.406498 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "04b1d19afced46e9b2a5dc77e913c059"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-h6n5"
server_key: "b3556027389f390400853cb290af8538"
server_key_iv: "42bc613e42f922722ae3dc118873e827"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:40.408398 11557 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:20:40.423760 11572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:40.424088 11557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:20:40.424206 11557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "04b1d19afced46e9b2a5dc77e913c059"
format_stamp: "Formatted at 2026-06-20 14:20:40 on dist-test-slave-h6n5"
server_key: "b3556027389f390400853cb290af8538"
server_key_iv: "42bc613e42f922722ae3dc118873e827"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:40.424312 11557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:40.446590 11557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:40.448957 11557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:40.449143 11557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:40.453516 11557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40793
I20260620 14:20:40.453584 11624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40793 every 8 connection(s)
I20260620 14:20:40.453953 11557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:20:40.454696 11625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:40.458259 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059: Bootstrap starting.
I20260620 14:20:40.458958 11625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:40.459239 11625 log.cc:824] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:40.460088 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059: No bootstrap required, opened a new log
I20260620 14:20:40.461221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 11557
I20260620 14:20:40.461349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:20:40.461540 392 external_mini_cluster.cc:1428] Setting key 997f4a0d12b5132e2aaf1698ba85af12
I20260620 14:20:40.462874 11625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b1d19afced46e9b2a5dc77e913c059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260620 14:20:40.463135 11625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:40.463192 11625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b1d19afced46e9b2a5dc77e913c059, State: Initialized, Role: FOLLOWER
I20260620 14:20:40.463402 11625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b1d19afced46e9b2a5dc77e913c059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260620 14:20:40.463496 11625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:20:40.463528 11625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:20:40.463630 11625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:40.464402 11625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b1d19afced46e9b2a5dc77e913c059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260620 14:20:40.464514 11625 leader_election.cc:302] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04b1d19afced46e9b2a5dc77e913c059; no voters:
I20260620 14:20:40.464682 11625 leader_election.cc:288] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:20:40.464725 11630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:40.464965 11630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [term 1 LEADER]: Becoming Leader. State: Replica: 04b1d19afced46e9b2a5dc77e913c059, State: Running, Role: LEADER
I20260620 14:20:40.465067 11625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:20:40.465114 11630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b1d19afced46e9b2a5dc77e913c059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260620 14:20:40.465507 11630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04b1d19afced46e9b2a5dc77e913c059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b1d19afced46e9b2a5dc77e913c059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } } }
I20260620 14:20:40.465638 11630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:40.465797 11630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b1d19afced46e9b2a5dc77e913c059. Latest consensus state: current_term: 1 leader_uuid: "04b1d19afced46e9b2a5dc77e913c059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b1d19afced46e9b2a5dc77e913c059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } } }
I20260620 14:20:40.465883 11630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059 [sys.catalog]: This master's current role is: LEADER
I20260620 14:20:40.466042 11639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:20:40.466559 11639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:20:40.466835 11625 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 14:20:40 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:40.475387 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:40.462093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.0.98.62:40793)
0620 14:20:40.462264 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:40.462275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:40.462424 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:20:40.462866 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:40.462874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:40.462929 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:40.463087 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:40.463877 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.465000 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.466127 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:40.466765 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:40.470043 (+ 3278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:40.470469 (+ 426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:40.470475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:40.470475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:40.470491 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:40.473151 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.473581 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.473584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.473584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.473700 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:40.473941 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:40.473941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:40.473942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:40.474305 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:40.474472 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:40.474584 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:20:40.475093 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 14:20:40.481389 11639 catalog_manager.cc:1347] Generated new cluster ID: f7ba017f8a684919b4041307658ac653
I20260620 14:20:40.481441 11639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:20:40.504412 11639 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:20:40.505241 11639 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:20:40.510933 11639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059: Generated new TSK 0
I20260620 14:20:40.511219 11639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:20:41.237746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:41.346051 11670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:41.346338 11670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:41.346376 11670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:41.348145 11670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:41.348217 11670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:41.348259 11670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:20:41.350489 11670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:20:41.350749 11670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:41.351043 11670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:41.356384 11678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:41.356391 11676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:41.356882 11675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:41.357463 11670 server_base.cc:1034] running on GCE node
I20260620 14:20:41.357679 11670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:41.357956 11670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:41.361244 11670 hybrid_clock.cc:648] HybridClock initialized: now 1781965241361211 us; error 43 us; skew 500 ppm
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:41.364449 11670 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:20:41.365052 11670 webserver.cc:460] Webserver started at http://127.0.98.1:39413/ using document root <none> and password file <none>
I20260620 14:20:41.365399 11670 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:41.365468 11670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:41.365612 11670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:20:41.380924 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "b00c1622a6974b1b98406382fcc1ab05"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "5bdecc6a2e02ddfb2d0f24d83b8fecde"
server_key_iv: "33dbe5c3397d0e447f1abad08f6b4815"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.381403 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b00c1622a6974b1b98406382fcc1ab05"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "5bdecc6a2e02ddfb2d0f24d83b8fecde"
server_key_iv: "33dbe5c3397d0e447f1abad08f6b4815"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.383965 11670 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:20:41.401432 11686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.403829 11670 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 14:20:41.404027 11670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "b00c1622a6974b1b98406382fcc1ab05"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "5bdecc6a2e02ddfb2d0f24d83b8fecde"
server_key_iv: "33dbe5c3397d0e447f1abad08f6b4815"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.404203 11670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:41.423316 11670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:41.425904 11670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:41.426120 11670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:41.426409 11670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:41.426832 11670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:41.426939 11670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.427044 11670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:41.427134 11670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.434999 11670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46415
I20260620 14:20:41.435532 11670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:20:41.436481 11800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46415 every 8 connection(s)
I20260620 14:20:41.443761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 11670
I20260620 14:20:41.443864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:20:41.444073 392 external_mini_cluster.cc:1428] Setting key 71f4e6400428f7d107250ef211a5c6f4
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:41.461232 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.440386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.1:48949)
0620 14:20:41.440598 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:41.440651 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:41.441288 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:41.441747 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:41.441753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:41.442002 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:41.442279 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.442285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.443326 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.443327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.444696 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.444699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.445197 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.445198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.445341 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.445956 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:41.445967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:41.447671 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:41.459368 (+ 11697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.459381 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.459383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.459780 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.460062 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.460063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.460064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.460211 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.460511 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:41.460519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:41.460717 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 14:20:41.460812 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:20:41.460989 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260620 14:20:41.461943 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.440694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48949 (local address 127.0.98.62:40793)
0620 14:20:41.441497 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:41.441499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:41.441509 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:20:41.441536 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:41.441537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:41.441561 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:41.442373 (+ 812us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:41.442414 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.443209 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.444802 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.445096 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.446645 (+ 1549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.451678 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:41.451680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:41.451681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:41.451689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:41.459231 (+ 7542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.459880 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.459881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.459881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.459971 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.460285 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.460286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.460286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.460427 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:41.461708 (+ 1281us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:41.461753 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:20:41.461842 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":71,"threads_started":1}
I20260620 14:20:41.462328 11801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260620 14:20:41.462445 11801 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:41.462638 11801 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260620 14:20:41.463230 11589 ts_manager.cc:194] Registered new tserver with Master: b00c1622a6974b1b98406382fcc1ab05 (127.0.98.1:46415)
I20260620 14:20:41.464183 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48949
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:20:41.493594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:41.595431 11807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:41.595727 11807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:41.595772 11807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:41.596956 11807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:41.597024 11807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:41.597093 11807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:20:41.598618 11807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:20:41.598883 11807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:41.599113 11807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:20:41.606168 11814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:41.606307 11817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:41.606647 11815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:41.607213 11807 server_base.cc:1034] running on GCE node
I20260620 14:20:41.607443 11807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:41.607750 11807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:41.608897 11807 hybrid_clock.cc:648] HybridClock initialized: now 1781965241608872 us; error 37 us; skew 500 ppm
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:41.611805 11807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:20:41.612308 11807 webserver.cc:460] Webserver started at http://127.0.98.2:33587/ using document root <none> and password file <none>
I20260620 14:20:41.612603 11807 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:41.612669 11807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:41.612813 11807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:20:41.625718 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "298bcce90962e6c93bc01b74a7776211"
server_key_iv: "54c6e02fd704b2965270307661f84641"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.626253 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "298bcce90962e6c93bc01b74a7776211"
server_key_iv: "54c6e02fd704b2965270307661f84641"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.630151 11807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:20:41.653657 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.653889 11807 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 14:20:41.653973 11807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "298bcce90962e6c93bc01b74a7776211"
server_key_iv: "54c6e02fd704b2965270307661f84641"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.654067 11807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:41.681051 11807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:41.683534 11807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:41.683722 11807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:41.683979 11807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:41.684360 11807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:41.684414 11807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.684456 11807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:41.684525 11807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.691612 11807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42587
I20260620 14:20:41.692112 11807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:20:41.694160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 11807
I20260620 14:20:41.694274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:20:41.694473 392 external_mini_cluster.cc:1428] Setting key 03a1e6c32348cce311ea315e8d5d483b
I20260620 14:20:41.700241 11938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42587 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:41.723882 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.703586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44089 (local address 127.0.98.62:40793)
0620 14:20:41.703637 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:41.703639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:41.707988 (+ 4349us) server_negotiation.cc:407] Connection header received
0620 14:20:41.708972 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:41.708974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:41.708986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:41.709016 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:41.710568 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.711376 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.712866 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.713187 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.713235 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.718740 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:41.718742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:41.718742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:41.718751 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:41.720559 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.721276 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.721278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.721278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.721339 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.723026 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.723027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.723028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.723179 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:41.723236 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:41.723671 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 14:20:41.723772 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:20:41.724521 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.703258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.2:44089)
0620 14:20:41.707823 (+ 4565us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:41.707881 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:41.708831 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:41.709272 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:41.709276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:41.710132 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:41.710450 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.710457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.711499 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.711500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.712762 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.712765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.713333 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.713334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.713488 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.716855 (+ 3367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:41.716867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:41.718614 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:41.720845 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.720852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.720855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.721171 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.721406 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.721407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.721407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.721539 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.723271 (+ 1732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:41.723277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:41.723475 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 14:20:41.724095 (+ 620us) client_negotiation.cc:240] Negotiation successful
0620 14:20:41.724268 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4510,"thread_start_us":93,"threads_started":1}
I20260620 14:20:41.725126 11939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260620 14:20:41.725226 11939 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:41.725432 11939 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260620 14:20:41.725910 11589 ts_manager.cc:194] Registered new tserver with Master: 248d4b77f9c94b6a9d9e8a3a0ba806fc (127.0.98.2:42587)
I20260620 14:20:41.726398 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:20:41.742235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:20:41.869745 11944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:20:41.870132 11944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:20:41.870257 11944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:20:41.872162 11944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:20:41.872396 11944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:20:41.872501 11944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:20:41.874881 11944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:20:41.875149 11944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:20:41.875480 11944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:20:41.882121 11952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:41.883671 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:20:41.884631 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:20:41.885145 11944 server_base.cc:1034] running on GCE node
I20260620 14:20:41.885387 11944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:20:41.886157 11944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:20:41.887348 11944 hybrid_clock.cc:648] HybridClock initialized: now 1781965241887316 us; error 45 us; skew 500 ppm
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:20:41.890792 11944 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:20:41.891431 11944 webserver.cc:460] Webserver started at http://127.0.98.3:41487/ using document root <none> and password file <none>
I20260620 14:20:41.891775 11944 fs_manager.cc:359] Metadata directory not provided
I20260620 14:20:41.891867 11944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:20:41.892082 11944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:20:41.911864 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ca4acecf86944f6a758362c41293883"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "8c0ae5cffd2129091247047ce07da019"
server_key_iv: "79ce39bcd3681d1bfc004848a28b662c"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.912818 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ca4acecf86944f6a758362c41293883"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "8c0ae5cffd2129091247047ce07da019"
server_key_iv: "79ce39bcd3681d1bfc004848a28b662c"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.915431 11944 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:20:41.940311 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.940593 11944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:20:41.940711 11944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "9ca4acecf86944f6a758362c41293883"
format_stamp: "Formatted at 2026-06-20 14:20:41 on dist-test-slave-h6n5"
server_key: "8c0ae5cffd2129091247047ce07da019"
server_key_iv: "79ce39bcd3681d1bfc004848a28b662c"
server_key_version: "kuduclusterkey@0"
I20260620 14:20:41.940811 11944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:20:41.954864 11944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:20:41.973426 11944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:20:41.973659 11944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:20:41.973991 11944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:20:41.974443 11944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:20:41.974531 11944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.974606 11944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:20:41.974663 11944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:20:41.981683 11944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38709
I20260620 14:20:41.982239 11944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:20:41.982537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 11944
I20260620 14:20:41.982651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:20:41.982903 392 external_mini_cluster.cc:1428] Setting key a620cfe5d70b0323386d2e56ca578a33
I20260620 14:20:41.985431 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38709 every 8 connection(s)
Jun 20 14:20:41 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:20:41.997468 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.984667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42945 (local address 127.0.98.62:40793)
0620 14:20:41.984711 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:41.984713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:41.986021 (+ 1308us) server_negotiation.cc:407] Connection header received
0620 14:20:41.986682 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:41.986684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:41.986695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:41.986724 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:41.987537 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.988265 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.989758 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.990045 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.990098 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.992787 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:41.992789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:41.992789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:41.992797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:41.995704 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.996234 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.996235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.996235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.996303 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:41.996622 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:41.996623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:41.996624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:41.996776 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:41.996826 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:41.997286 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 14:20:41.997376 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:20:41.997659 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:41.984379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.3:42945)
0620 14:20:41.985747 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:41.985921 (+ 174us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:41.986564 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:41.986902 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:41.986906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:41.987165 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:20:41.987433 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.987439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.988387 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.988388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.989653 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:41.989656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:41.990190 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:41.990191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:41.990324 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:41.990961 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:20:41.990971 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:20:41.992681 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:20:41.995804 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.995812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.995815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.996135 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.996378 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:20:41.996379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:20:41.996379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:20:41.996528 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:20:41.996878 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:20:41.996885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:20:41.997079 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 14:20:41.997219 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 14:20:41.997401 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"thread_start_us":98,"threads_started":1}
I20260620 14:20:41.998265 12073 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260620 14:20:41.998378 12073 heartbeater.cc:464] Registering TS with master...
I20260620 14:20:41.998610 12073 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260620 14:20:41.999082 11589 ts_manager.cc:194] Registered new tserver with Master: 9ca4acecf86944f6a758362c41293883 (127.0.98.3:38709)
I20260620 14:20:41.999537 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42945
I20260620 14:20:42.005237 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:20:42.016889 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.006852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.0.98.62:40793)
0620 14:20:42.006892 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.006894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.007154 (+ 260us) server_negotiation.cc:407] Connection header received
0620 14:20:42.007245 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.007245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.007255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.007283 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:20:42.008835 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.009568 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.010788 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.011069 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.011107 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.013155 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:20:42.013156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:20:42.013156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:20:42.013164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:20:42.015511 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:42.015969 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:42.015970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:42.015971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:42.016033 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:20:42.016265 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:20:42.016266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:20:42.016266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:20:42.016400 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:20:42.016476 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.016702 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.016794 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:20:42.019217 11589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:20:42.021662 11589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:20:42.035935 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.025980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46415 (local address 127.0.0.1:32988)
0620 14:20:42.027177 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:42.027185 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:42.027230 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:42.027965 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:42.027965 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:42.027967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:42.027998 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.027999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.030984 (+ 2985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.030986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.032340 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.032344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.035607 (+ 3263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.035608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.035692 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.035710 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:20:42.035789 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:20:42.035844 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"thread_start_us":65,"threads_started":1}
I20260620 14:20:42.036415 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.026321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.0.98.1:46415)
0620 14:20:42.027127 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.027132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.027279 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:20:42.027316 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.027319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.027335 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.027402 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:42.028072 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.030900 (+ 2828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.032450 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.033207 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.033279 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.033344 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.036285 (+ 2941us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.036339 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":47,"threads_started":1}
I20260620 14:20:42.038064 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.025295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42587 (local address 127.0.0.1:36612)
0620 14:20:42.026740 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:42.026784 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:42.026851 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:42.027810 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:42.027814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:42.027817 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:42.027887 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.027889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.034040 (+ 6151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.034041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.035159 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.035162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.037818 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.037819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.037884 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.037889 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:20:42.037940 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:20:42.037982 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1414,"thread_start_us":73,"threads_started":1}
I20260620 14:20:42.038870 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.025565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.0.98.2:42587)
0620 14:20:42.027533 (+ 1968us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.027538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.027583 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:20:42.027627 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.027629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.027644 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.027719 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:42.029714 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.033944 (+ 4230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.035328 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.037628 (+ 2300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.037702 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.037781 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.038733 (+ 952us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.038780 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":78,"threads_started":1}
I20260620 14:20:42.038898 11735 tablet_service.cc:1467] Processing CreateTablet for tablet bfb2205ac5af493f86a4e4094e218aa7 (DEFAULT_TABLE table=test-table [id=c6e31a95dd5c42b793afbc96eb5f27d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:42.039310 11735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb2205ac5af493f86a4e4094e218aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:42.039690 11857 tablet_service.cc:1467] Processing CreateTablet for tablet bfb2205ac5af493f86a4e4094e218aa7 (DEFAULT_TABLE table=test-table [id=c6e31a95dd5c42b793afbc96eb5f27d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:42.040042 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb2205ac5af493f86a4e4094e218aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:42.047714 12093 tablet_bootstrap.cc:492] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Bootstrap starting.
I20260620 14:20:42.048869 12093 tablet_bootstrap.cc:654] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:42.049263 12093 log.cc:824] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:42.055755 12092 tablet_bootstrap.cc:492] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: Bootstrap starting.
I20260620 14:20:42.056217 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.025818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38709 (local address 127.0.0.1:45016)
0620 14:20:42.026960 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:42.026969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:42.027012 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:42.051921 (+ 24909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:42.051922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:42.051926 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:42.051990 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.051993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.053349 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.053350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.054511 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.054514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.055980 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.055981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.056052 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.056058 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:20:42.056087 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:20:42.056133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":72,"threads_started":1}
I20260620 14:20:42.056744 12092 tablet_bootstrap.cc:654] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:42.056828 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.026191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.0.98.3:38709)
0620 14:20:42.051711 (+ 25520us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.051717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.051730 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:20:42.051774 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.051776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.051793 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.052063 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:42.052142 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.053238 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.054651 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.055879 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.056568 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.056679 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.056706 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.056747 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25473,"thread_start_us":84,"threads_started":1}
I20260620 14:20:42.057497 12093 tablet_bootstrap.cc:492] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: No bootstrap required, opened a new log
I20260620 14:20:42.057653 12093 ts_tablet_manager.cc:1397] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260620 14:20:42.057770 11992 tablet_service.cc:1467] Processing CreateTablet for tablet bfb2205ac5af493f86a4e4094e218aa7 (DEFAULT_TABLE table=test-table [id=c6e31a95dd5c42b793afbc96eb5f27d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:20:42.058113 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb2205ac5af493f86a4e4094e218aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:20:42.060029 12093 raft_consensus.cc:348] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.060316 12093 raft_consensus.cc:374] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:42.060420 12093 raft_consensus.cc:729] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248d4b77f9c94b6a9d9e8a3a0ba806fc, State: Initialized, Role: FOLLOWER
I20260620 14:20:42.060619 12093 consensus_queue.cc:260] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.060999 12093 ts_tablet_manager.cc:1428] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:20:42.061170 11939 heartbeater.cc:503] Master 127.0.98.62:40793 was elected leader, sending a full tablet report...
I20260620 14:20:42.061990 12092 log.cc:824] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:42.062302 12096 tablet_bootstrap.cc:492] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Bootstrap starting.
I20260620 14:20:42.063635 12096 tablet_bootstrap.cc:654] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Neither blocks nor log segments found. Creating new log.
I20260620 14:20:42.063975 12096 log.cc:824] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Log is configured to *not* fsync() on all Append() calls
I20260620 14:20:42.064834 12092 tablet_bootstrap.cc:492] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: No bootstrap required, opened a new log
I20260620 14:20:42.064960 12092 ts_tablet_manager.cc:1397] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 14:20:42.068151 12092 raft_consensus.cc:348] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.068387 12092 raft_consensus.cc:374] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:42.068466 12092 raft_consensus.cc:729] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00c1622a6974b1b98406382fcc1ab05, State: Initialized, Role: FOLLOWER
I20260620 14:20:42.068682 12092 consensus_queue.cc:260] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.069012 12092 ts_tablet_manager.cc:1428] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 14:20:42.069175 11801 heartbeater.cc:503] Master 127.0.98.62:40793 was elected leader, sending a full tablet report...
I20260620 14:20:42.070547 12096 tablet_bootstrap.cc:492] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: No bootstrap required, opened a new log
I20260620 14:20:42.070686 12096 ts_tablet_manager.cc:1397] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260620 14:20:42.073107 12096 raft_consensus.cc:348] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.073328 12096 raft_consensus.cc:374] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:20:42.073414 12096 raft_consensus.cc:729] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca4acecf86944f6a758362c41293883, State: Initialized, Role: FOLLOWER
I20260620 14:20:42.073628 12096 consensus_queue.cc:260] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.073944 12096 ts_tablet_manager.cc:1428] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 14:20:42.074108 12073 heartbeater.cc:503] Master 127.0.98.62:40793 was elected leader, sending a full tablet report...
Jun 20 14:20:42 dist-test-slave-h6n5 krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:20:42.178227 12099 raft_consensus.cc:482] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:20:42.178377 12099 raft_consensus.cc:504] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.178763 12099 leader_election.cc:288] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca4acecf86944f6a758362c41293883 (127.0.98.3:38709), 248d4b77f9c94b6a9d9e8a3a0ba806fc (127.0.98.2:42587)
I20260620 14:20:42.182997 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.179164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39373 (local address 127.0.98.3:38709)
0620 14:20:42.179210 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.179212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.179452 (+ 240us) server_negotiation.cc:407] Connection header received
0620 14:20:42.179494 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.179496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.179509 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.179618 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:42.179853 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.180613 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.182081 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.182611 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.182662 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.182673 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.182888 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.182927 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:20:42.183261 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.178937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38709 (local address 127.0.98.1:39373)
0620 14:20:42.179363 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:42.179374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:42.179441 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:42.179677 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:42.179679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:42.179681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:42.179767 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.179769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.180743 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.180744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.181971 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.181975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.182735 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.182736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.182804 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.182808 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:20:42.183132 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 14:20:42.183179 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":116,"threads_started":1}
I20260620 14:20:42.183617 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb2205ac5af493f86a4e4094e218aa7" candidate_uuid: "b00c1622a6974b1b98406382fcc1ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca4acecf86944f6a758362c41293883" is_pre_election: true
I20260620 14:20:42.183775 12027 raft_consensus.cc:2454] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00c1622a6974b1b98406382fcc1ab05 in term 0.
I20260620 14:20:42.184002 11691 leader_election.cc:302] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca4acecf86944f6a758362c41293883, b00c1622a6974b1b98406382fcc1ab05; no voters:
I20260620 14:20:42.184139 12099 raft_consensus.cc:2781] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:20:42.184175 12099 raft_consensus.cc:482] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:20:42.184196 12099 raft_consensus.cc:3037] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:42.185029 12099 raft_consensus.cc:504] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.185168 12099 leader_election.cc:288] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [CANDIDATE]: Term 1 election: Requested vote from peers 9ca4acecf86944f6a758362c41293883 (127.0.98.3:38709), 248d4b77f9c94b6a9d9e8a3a0ba806fc (127.0.98.2:42587)
I20260620 14:20:42.185439 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb2205ac5af493f86a4e4094e218aa7" candidate_uuid: "b00c1622a6974b1b98406382fcc1ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca4acecf86944f6a758362c41293883"
I20260620 14:20:42.185520 12027 raft_consensus.cc:3037] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:20:42.186313 12027 raft_consensus.cc:2454] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00c1622a6974b1b98406382fcc1ab05 in term 1.
I20260620 14:20:42.186487 11691 leader_election.cc:302] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca4acecf86944f6a758362c41293883, b00c1622a6974b1b98406382fcc1ab05; no voters:
I20260620 14:20:42.186566 12099 raft_consensus.cc:2781] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:20:42.186703 12099 raft_consensus.cc:686] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 1 LEADER]: Becoming Leader. State: Replica: b00c1622a6974b1b98406382fcc1ab05, State: Running, Role: LEADER
I20260620 14:20:42.186796 12099 consensus_queue.cc:237] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } }
I20260620 14:20:42.187587 11589 catalog_manager.cc:5495] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 reported cstate change: term changed from 0 to 1, leader changed from <none> to b00c1622a6974b1b98406382fcc1ab05 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b00c1622a6974b1b98406382fcc1ab05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b00c1622a6974b1b98406382fcc1ab05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46415 } health_report { overall_health: HEALTHY } } }
I20260620 14:20:42.190738 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.179170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33065 (local address 127.0.98.2:42587)
0620 14:20:42.183626 (+ 4456us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.183629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.183638 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:20:42.183666 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.183667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.183681 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.183740 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:20:42.187796 (+ 4056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.188433 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.189793 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.190344 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.190391 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.190401 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.190625 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.190670 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4424}
I20260620 14:20:42.190963 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.178944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42587 (local address 127.0.98.1:33065)
0620 14:20:42.179514 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:42.179521 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:20:42.179571 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:20:42.187631 (+ 8060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:20:42.187632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:20:42.187635 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:20:42.187690 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.187691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.188560 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.188561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.189683 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:20:42.189686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.190460 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.190461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:20:42.190527 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.190531 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:20:42.190825 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 14:20:42.190865 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":121,"threads_started":1}
I20260620 14:20:42.191347 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb2205ac5af493f86a4e4094e218aa7" candidate_uuid: "b00c1622a6974b1b98406382fcc1ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" is_pre_election: true
I20260620 14:20:42.191498 11893 raft_consensus.cc:2454] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00c1622a6974b1b98406382fcc1ab05 in term 0.
I20260620 14:20:42.191638 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb2205ac5af493f86a4e4094e218aa7" candidate_uuid: "b00c1622a6974b1b98406382fcc1ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc"
I20260620 14:20:42.191699 11892 raft_consensus.cc:3037] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:20:42.192019 11802 tablet.cc:2369] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:42.192464 11892 raft_consensus.cc:2454] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00c1622a6974b1b98406382fcc1ab05 in term 1.
W20260620 14:20:42.203054 11940 tablet.cc:2369] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:20:42.212383 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.209204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.0.98.1:46415)
0620 14:20:42.209255 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:20:42.209257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:20:42.209267 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:20:42.209329 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:20:42.209330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:20:42.209353 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:20:42.209417 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:20:42.209566 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.210326 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.211368 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:20:42.211688 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:20:42.211758 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:20:42.211933 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:20:42.212101 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:20:42.212184 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:20:42.212261 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:20:42.212284 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:20:42.214284 11892 raft_consensus.cc:1261] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 1 FOLLOWER]: Refusing update from remote peer b00c1622a6974b1b98406382fcc1ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:42.214609 12027 raft_consensus.cc:1261] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 1 FOLLOWER]: Refusing update from remote peer b00c1622a6974b1b98406382fcc1ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:20:42.214936 12103 consensus_queue.cc:1035] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca4acecf86944f6a758362c41293883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:42.215107 12103 consensus_queue.cc:1035] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248d4b77f9c94b6a9d9e8a3a0ba806fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:20:42.218825 12106 mvcc.cc:205] Tried to move back new op lower bound from 7298929632106999808 to 7298929631997612032. Current Snapshot: MvccSnapshot[applied={T|T < 7298929632106999808}]
I20260620 14:20:42.218937 12107 mvcc.cc:205] Tried to move back new op lower bound from 7298929632106999808 to 7298929631997612032. Current Snapshot: MvccSnapshot[applied={T|T < 7298929632106999808}]
I20260620 14:20:42.219120 12108 mvcc.cc:205] Tried to move back new op lower bound from 7298929632106999808 to 7298929631997612032. Current Snapshot: MvccSnapshot[applied={T|T < 7298929632106999808}]
I20260620 14:20:42.221563 11588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:42.221659 11588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:20:42.222429 11588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04b1d19afced46e9b2a5dc77e913c059: Sending DeleteTablet for 3 replicas of tablet bfb2205ac5af493f86a4e4094e218aa7
I20260620 14:20:42.222741 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb2205ac5af493f86a4e4094e218aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36612
I20260620 14:20:42.223170 11992 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb2205ac5af493f86a4e4094e218aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45016
I20260620 14:20:42.223780 11735 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb2205ac5af493f86a4e4094e218aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:20:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32988
I20260620 14:20:42.225046 12116 tablet_replica.cc:330] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: stopping tablet replica
I20260620 14:20:42.225194 12116 raft_consensus.cc:2229] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:20:42.225356 12116 raft_consensus.cc:2258] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:42.225526 12115 tablet_replica.cc:330] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: stopping tablet replica
I20260620 14:20:42.225666 12117 tablet_replica.cc:330] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: stopping tablet replica
I20260620 14:20:42.225801 12115 raft_consensus.cc:2229] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:42.225862 12116 ts_tablet_manager.cc:1905] T bfb2205ac5af493f86a4e4094e218aa7 P b00c1622a6974b1b98406382fcc1ab05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:42.225924 12117 raft_consensus.cc:2229] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:20:42.226176 12115 raft_consensus.cc:2258] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:42.226317 12117 raft_consensus.cc:2258] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:20:42.226824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 11670
I20260620 14:20:42.227545 12117 ts_tablet_manager.cc:1905] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:42.229090 12115 ts_tablet_manager.cc:1905] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:20:42.229326 12117 ts_tablet_manager.cc:1918] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:42.229389 12117 log.cc:1196] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/bfb2205ac5af493f86a4e4094e218aa7
I20260620 14:20:42.229993 12117 ts_tablet_manager.cc:1939] T bfb2205ac5af493f86a4e4094e218aa7 P 9ca4acecf86944f6a758362c41293883: Deleting consensus metadata
I20260620 14:20:42.230435 11576 catalog_manager.cc:4841] TS 9ca4acecf86944f6a758362c41293883 (127.0.98.3:38709): tablet bfb2205ac5af493f86a4e4094e218aa7 (table test-table [id=c6e31a95dd5c42b793afbc96eb5f27d6]) successfully deleted
I20260620 14:20:42.230923 12115 ts_tablet_manager.cc:1918] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:20:42.230996 12115 log.cc:1196] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/bfb2205ac5af493f86a4e4094e218aa7
I20260620 14:20:42.231235 12115 ts_tablet_manager.cc:1939] T bfb2205ac5af493f86a4e4094e218aa7 P 248d4b77f9c94b6a9d9e8a3a0ba806fc: Deleting consensus metadata
I20260620 14:20:42.231690 11573 catalog_manager.cc:4841] TS 248d4b77f9c94b6a9d9e8a3a0ba806fc (127.0.98.2:42587): tablet bfb2205ac5af493f86a4e4094e218aa7 (table test-table [id=c6e31a95dd5c42b793afbc96eb5f27d6]) successfully deleted
W20260620 14:20:42.233788 11576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46415 (error 108)
I20260620 14:20:42.234354 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:20:42.234235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46415 (local address 127.0.0.1:33010)
0620 14:20:42.234284 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:20:42.234319 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:20:42.234473 11576 catalog_manager.cc:4584] TS b00c1622a6974b1b98406382fcc1ab05 (127.0.98.1:46415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfb2205ac5af493f86a4e4094e218aa7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46415: connect: Connection refused (error 111)
I20260620 14:20:42.235714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 11807
I20260620 14:20:42.242645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 11944
I20260620 14:20:42.248488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 11557
I20260620 14:20:42.253551 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:20:42.588320 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:20:42.588428 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:20:42.933182 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:20:42.933 UTC [11143] LOG: received smart shutdown request
2026-06-20 14:20:42.936 UTC [11143] LOG: background worker "logical replication launcher" (PID 11152) exited with exit code 1
2026-06-20 14:20:42.936 UTC [11147] LOG: shutting down
2026-06-20 14:20:43.096 UTC [11143] LOG: database system is shut down
2026-06-20T14:20:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45724 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:20:43 dist-test-slave-h6n5 krb5kdc[12127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:20:43 dist-test-slave-h6n5 krb5kdc[12127](info): set up 2 sockets
Jun 20 14:20:43 dist-test-slave-h6n5 krb5kdc[12127](info): commencing operation
krb5kdc: starting...
W20260620 14:20:44.247406 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:20:44 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:20:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:20:44.404227 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTTnugX/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:20:46.890 UTC [12157] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:20:46.890 UTC [12157] LOG: listening on IPv4 address "127.0.98.20", port 50945
2026-06-20 14:20:46.893 UTC [12157] LOG: listening on Unix socket "/tmp/.s.PGSQL.50945"
2026-06-20 14:20:46.914 UTC [12160] LOG: database system was shut down at 2026-06-20 14:20:45 UTC
2026-06-20 14:20:46.918 UTC [12157] LOG: database system is ready to accept connections
I20260620 14:20:47.871484 392 mini_postgres.cc:96] Postgres bound to 50945
2026-06-20 14:20:47.877 UTC [12168] FATAL: database "slave" does not exist
127.0.98.20:50945 - accepting connections
I20260620 14:20:47.878008 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:20:47.892999 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:20:48.363040 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:20:48.363131 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-admin
2026-06-20 14:20:48,642 [I] DB FLAVOR :POSTGRES
2026-06-20 14:20:48,643 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:20:48,643 [I] Checking connection..
2026-06-20 14:20:48,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:20:48,899 [I] Checking connection passed.
2026-06-20 14:20:48,899 [I] --------- Verifying version history table ---------
2026-06-20 14:20:48,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:20:49,161 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:20:49,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:20:49,419 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:20:49,419 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:20:49,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:20:49.650 UTC [12232] WARNING: there is no transaction in progress
2026-06-20 14:20:49,657 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:20:49,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:20:49,930 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:20:49,930 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:20:49,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:20:50,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:20:50,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h6n5.c.gcp-upstream.internal','N') ;"
2026-06-20 14:20:50,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:20:50,929 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:20:50,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:20:51,166 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:20:51,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:20:51,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:20:51,637 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:20:51,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:20:52.580 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.597 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.615 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.631 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.647 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.667 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.734 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.742 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.757 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.767 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.777 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.791 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.800 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.807 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.814 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:52.821 UTC [12340] WARNING: there is no transaction in progress
2026-06-20 14:20:53,468 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:20:53,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:20:53,724 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:20:53,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:20:53,974 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:20:53,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:20:54,235 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:20:54,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:20:54,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:20:54,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h6n5.c.gcp-upstream.internal';"
2026-06-20 14:20:54,978 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:20:54,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:20:55,230 [I] DB_PATCHES have already been applied
I20260620 14:20:55.235742 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTTnugX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:20:55.235813 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:20:55.238826 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58275
Jun 20, 2026 2:20:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:20:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:20:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:20:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:20:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:20:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:20:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:20:56.362111 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.484s user 0.000s sys 0.009s
Jun 20, 2026 2:21:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:21:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 2:21:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:21:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:21:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:21:18.320772 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:21:18 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965278, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:21:18.403893 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:21:18.412859 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:21:19.278115 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:21:19.278208 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms
I20260620 14:21:19.278241 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50945
2026-06-20 14:21:19,438 [I] DB FLAVOR :POSTGRES
2026-06-20 14:21:19,438 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:21:19,438 [I] Checking connection
2026-06-20 14:21:19,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:21:19,670 [I] connection success
2026-06-20 14:21:19,670 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:21:19,670 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:21:19,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:21:19,905 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:21:19,905 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:21:19,906 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:21:19,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:21:20,173 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:21:20.689982 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:21:20.893327 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:21:20.937067 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:21:20 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965278, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:21.362795 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:21:21.363013 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTTnugX/build/release/bin/postgresql.jar:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTTnugX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:21:21.363039 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:21:21.366374 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12520
I20260620 14:21:21.366459 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55841
Jun 20, 2026 2:21:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:21:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:21:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:21:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:21:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:21:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:21:23.630074 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.226s user 0.001s sys 0.141s
I20260620 14:21:23.630187 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:21:23.630230 392 mini_ranger_kms.cc:401] 127.0.98.20:55841/kms/v1/keys
Jun 20 14:21:23 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965283, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:21:24 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965283, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:21:24 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965283, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:21:26 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:21:26.994419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39509
--ranger_config_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:27.076290 12571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:27.076542 12571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:27.076582 12571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:27.077955 12571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:27.078022 12571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:27.078045 12571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:27.078066 12571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:27.078095 12571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:27.079746 12571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55841/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39509
--ranger_config_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:27.080021 12571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:27.080258 12571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:27.085760 12577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:27.085686 12571 server_base.cc:1034] running on GCE node
W20260620 14:21:27.085646 12576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:27.085700 12579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:27.086248 12571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:27.086611 12571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:27.087785 12571 hybrid_clock.cc:648] HybridClock initialized: now 1781965287087765 us; error 35 us; skew 500 ppm
Jun 20 14:21:27 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:27.094208 12571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:27.094708 12571 webserver.cc:460] Webserver started at http://127.0.98.62:43295/ using document root <none> and password file <none>
I20260620 14:21:27.094951 12571 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:27.095016 12571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:27.095183 12571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:21:27 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:27.143952 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "c1941a9f3fee436791e5b0bab51be078"
format_stamp: "Formatted at 2026-06-20 14:21:27 on dist-test-slave-h6n5"
server_key: "b59799781a33e9b54e9c7b678446298c"
server_key_iv: "f414fd5df116db14c7d9e8f0c9251357"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:27.144384 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1941a9f3fee436791e5b0bab51be078"
format_stamp: "Formatted at 2026-06-20 14:21:27 on dist-test-slave-h6n5"
server_key: "b59799781a33e9b54e9c7b678446298c"
server_key_iv: "f414fd5df116db14c7d9e8f0c9251357"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:27.145838 12571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:21:27.160400 12586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:27.160673 12571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:27.160769 12571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "c1941a9f3fee436791e5b0bab51be078"
format_stamp: "Formatted at 2026-06-20 14:21:27 on dist-test-slave-h6n5"
server_key: "b59799781a33e9b54e9c7b678446298c"
server_key_iv: "f414fd5df116db14c7d9e8f0c9251357"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:27.160856 12571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:27.190614 12571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:27.192956 12571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:27.193102 12571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:27.197916 12571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39509
I20260620 14:21:27.198002 12638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39509 every 8 connection(s)
I20260620 14:21:27.198348 12571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:27.199071 12639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:27.200933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 12571
I20260620 14:21:27.201023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:27.201233 392 external_mini_cluster.cc:1428] Setting key 9fbdb3523019c39f64b6514dae6c03a6
I20260620 14:21:27.201777 12639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Bootstrap starting.
I20260620 14:21:27.202442 12639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:27.202800 12639 log.cc:824] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:27.203651 12639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: No bootstrap required, opened a new log
I20260620 14:21:27.205547 12639 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:27.205718 12639 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:27.205751 12639 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1941a9f3fee436791e5b0bab51be078, State: Initialized, Role: FOLLOWER
I20260620 14:21:27.205920 12639 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:27.205991 12639 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:27.206018 12639 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:27.206074 12639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:27.206640 12639 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:27.206769 12639 leader_election.cc:302] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1941a9f3fee436791e5b0bab51be078; no voters:
I20260620 14:21:27.206945 12639 leader_election.cc:288] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:27.206974 12644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:27.207273 12639 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:27.208153 12644 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 LEADER]: Becoming Leader. State: Replica: c1941a9f3fee436791e5b0bab51be078, State: Running, Role: LEADER
I20260620 14:21:27.208333 12644 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:27.208880 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1941a9f3fee436791e5b0bab51be078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } } }
I20260620 14:21:27.208966 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:27.208994 12639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:21:27.209262 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1941a9f3fee436791e5b0bab51be078. Latest consensus state: current_term: 1 leader_uuid: "c1941a9f3fee436791e5b0bab51be078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } } }
I20260620 14:21:27.209316 12658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:27.209867 12658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:27.210129 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:21:27 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:27.219630 12658 catalog_manager.cc:1347] Generated new cluster ID: cd24bfcc9ba3413d8dadd0238b2ee868
I20260620 14:21:27.219700 12658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:27.221407 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:27.201787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.0.98.62:39509)
0620 14:21:27.202634 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:27.202645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:27.202682 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:21:27.203204 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:27.203213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:27.203263 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:27.203455 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:27.204273 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:27.206163 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:27.207223 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:27.207937 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:27.208057 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:27.210962 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:27.210966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:27.210967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:27.210979 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:27.218440 (+ 7461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:27.218929 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:27.218932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:27.218933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:27.219098 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:27.219356 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:27.219357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:27.219358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:27.219797 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:27.220381 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:27.221010 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 14:21:27.221160 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":71,"threads_started":1}
I20260620 14:21:27.238644 12658 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:27.239431 12658 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:27.247220 12658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Generated new TSK 0
I20260620 14:21:27.247467 12658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:21:27.995464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39509
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:28.085893 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:28.086200 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:28.086299 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:28.087541 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:28.087662 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:28.087709 12684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:28.089236 12684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:28.089515 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:28.089779 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:28.094950 12689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.094957 12692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.095103 12690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:28.095355 12684 server_base.cc:1034] running on GCE node
I20260620 14:21:28.095625 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:28.095901 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:28.097077 12684 hybrid_clock.cc:648] HybridClock initialized: now 1781965288097062 us; error 35 us; skew 500 ppm
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:28.099777 12684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:28.100217 12684 webserver.cc:460] Webserver started at http://127.0.98.1:42113/ using document root <none> and password file <none>
I20260620 14:21:28.100483 12684 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:28.100555 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:28.100724 12684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:28.123965 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "140110b0397c4d8297777d947426eb1c"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "905c44c52a740cde341b4122a9828f2d"
server_key_iv: "a9797d0d8a78a84aa51c25aacee372ec"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.124548 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "140110b0397c4d8297777d947426eb1c"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "905c44c52a740cde341b4122a9828f2d"
server_key_iv: "a9797d0d8a78a84aa51c25aacee372ec"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.126479 12684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:21:28.142982 12699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.143208 12684 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260620 14:21:28.143339 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "140110b0397c4d8297777d947426eb1c"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "905c44c52a740cde341b4122a9828f2d"
server_key_iv: "a9797d0d8a78a84aa51c25aacee372ec"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.143450 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:28.163334 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:28.165731 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:28.165902 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:28.166178 12684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:28.166558 12684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:28.166615 12684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.166682 12684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:28.166716 12684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.173887 12684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44917
I20260620 14:21:28.173939 12812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44917 every 8 connection(s)
I20260620 14:21:28.174535 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:28.179337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 12684
I20260620 14:21:28.179461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:21:28.179688 392 external_mini_cluster.cc:1428] Setting key ba766eef005e26f41e316b0883a8a507
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:21:28.189857 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:28.175363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60417 (local address 127.0.98.62:39509)
0620 14:21:28.175636 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:28.175638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:28.175784 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:21:28.176345 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:28.176347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:28.176364 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:28.176431 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:28.177026 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.177807 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.182314 (+ 4507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.182519 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.182644 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:28.186358 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:28.186360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:28.186360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:28.186368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:28.188239 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:28.188725 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:28.188727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:28.188727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:28.188792 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:28.189110 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:28.189111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:28.189111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:28.189238 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:28.189326 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:28.189663 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 14:21:28.189750 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
I20260620 14:21:28.190055 12815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:28.175347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39509 (local address 127.0.98.1:60417)
0620 14:21:28.175538 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:28.175688 (+ 150us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:28.176214 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:28.176504 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:28.176509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:28.176714 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:28.176912 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:28.176917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.177914 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.177915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:28.182192 (+ 4277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:28.182198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.182671 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.182672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:28.182886 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:28.183497 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:28.183505 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:28.186236 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:28.188369 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:28.188378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:28.188381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:28.188632 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:28.188893 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:28.188894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:28.188894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:28.189030 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:28.189311 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:28.189318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:28.189523 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 14:21:28.189629 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:21:28.189787 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":120,"threads_started":1}
I20260620 14:21:28.190713 12813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39509
I20260620 14:21:28.190817 12813 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:28.191032 12813 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:28.191581 12603 ts_manager.cc:194] Registered new tserver with Master: 140110b0397c4d8297777d947426eb1c (127.0.98.1:44917)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:21:28.192466 12603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:21:28.211933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39509
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:28.324738 12821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:28.325002 12821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:28.325035 12821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:28.326717 12821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:28.326776 12821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:28.326812 12821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:28.328922 12821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:28.329157 12821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:28.329461 12821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:28.335718 12828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.335712 12831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.335712 12829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:28.336766 12821 server_base.cc:1034] running on GCE node
I20260620 14:21:28.336964 12821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:28.337221 12821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:28.338406 12821 hybrid_clock.cc:648] HybridClock initialized: now 1781965288338372 us; error 46 us; skew 500 ppm
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:28.341778 12821 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:28.342342 12821 webserver.cc:460] Webserver started at http://127.0.98.2:37035/ using document root <none> and password file <none>
I20260620 14:21:28.342628 12821 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:28.342695 12821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:28.342875 12821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:28.356184 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "716025670b3844f0a8bf5ff3f46db52e"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "cd6d89ed3f4a41282b13659d2a352476"
server_key_iv: "06d598b5959ed95f329c31c8640e0fa7"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.356737 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "716025670b3844f0a8bf5ff3f46db52e"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "cd6d89ed3f4a41282b13659d2a352476"
server_key_iv: "06d598b5959ed95f329c31c8640e0fa7"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.360008 12821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:28.383908 12838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.384171 12821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 14:21:28.384280 12821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "716025670b3844f0a8bf5ff3f46db52e"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "cd6d89ed3f4a41282b13659d2a352476"
server_key_iv: "06d598b5959ed95f329c31c8640e0fa7"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.384377 12821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:28.408756 12821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:28.411648 12821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:28.411868 12821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:28.412133 12821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:28.412560 12821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:28.412612 12821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.412644 12821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:28.412663 12821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.419474 12821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43745
I20260620 14:21:28.419665 12951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43745 every 8 connection(s)
I20260620 14:21:28.419997 12821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:28.420488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 12821
I20260620 14:21:28.420603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:21:28.420831 392 external_mini_cluster.cc:1428] Setting key e747a3c715606b0201394fb7001f0e5c
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:28.436699 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:28.424135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40859 (local address 127.0.98.62:39509)
0620 14:21:28.424254 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:28.424256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:28.426058 (+ 1802us) server_negotiation.cc:407] Connection header received
0620 14:21:28.426835 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:28.426836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:28.426849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:28.426882 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:28.427863 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.428657 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.430226 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.430534 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.430585 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:28.433488 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:28.433489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:28.433490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:28.433498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:28.435154 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:28.435651 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:28.435653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:28.435653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:28.435708 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:28.435955 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:28.435956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:28.435956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:28.436092 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:28.436141 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:28.436501 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:21:28.436592 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:21:28.437237 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:28.420781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39509 (local address 127.0.98.2:40859)
0620 14:21:28.425887 (+ 5106us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:28.425954 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:28.426695 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:28.427096 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:28.427101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:28.427396 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:28.427726 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:28.427734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.428786 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.428788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:28.430098 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:28.430101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.430703 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.430704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:28.430851 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:28.431678 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:28.431689 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:28.433328 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:28.435253 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:28.435260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:28.435263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:28.435544 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:28.435769 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:28.435770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:28.435770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:28.435891 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:28.436172 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:28.436177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:28.436359 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:21:28.436834 (+ 475us) client_negotiation.cc:240] Negotiation successful
0620 14:21:28.436994 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5000,"thread_start_us":79,"threads_started":1}
I20260620 14:21:28.437700 12952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39509
I20260620 14:21:28.437803 12952 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:21:28.437988 12952 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:28.438333 12603 ts_manager.cc:194] Registered new tserver with Master: 716025670b3844f0a8bf5ff3f46db52e (127.0.98.2:43745)
I20260620 14:21:28.438820 12603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:21:28.459326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39509
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:28.567510 12957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:28.567833 12957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:28.567871 12957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:28.569572 12957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:28.569621 12957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:28.569670 12957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:28.571794 12957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:28.572052 12957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:28.572305 12957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:28.578792 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.578789 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.579118 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:28.580226 12957 server_base.cc:1034] running on GCE node
I20260620 14:21:28.580533 12957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:28.580845 12957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:28.582034 12957 hybrid_clock.cc:648] HybridClock initialized: now 1781965288582012 us; error 36 us; skew 500 ppm
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:28.585498 12957 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:28.586077 12957 webserver.cc:460] Webserver started at http://127.0.98.3:32781/ using document root <none> and password file <none>
I20260620 14:21:28.586359 12957 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:28.586433 12957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:28.586584 12957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:28.609352 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f3e736b55bf40369a2d722934a662f2"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "fa5183ac54d467f2ce6fe2c5404ba800"
server_key_iv: "37c5d549dd10252e05162070128fb40a"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.610484 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f3e736b55bf40369a2d722934a662f2"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "fa5183ac54d467f2ce6fe2c5404ba800"
server_key_iv: "37c5d549dd10252e05162070128fb40a"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.613346 12957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:21:28.628237 12973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.628440 12957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:21:28.628535 12957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "3f3e736b55bf40369a2d722934a662f2"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "fa5183ac54d467f2ce6fe2c5404ba800"
server_key_iv: "37c5d549dd10252e05162070128fb40a"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.628635 12957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:28.647475 12957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:28.649641 12957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:28.649801 12957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:28.650014 12957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:28.650393 12957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:28.650454 12957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.650489 12957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:28.650506 12957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.658378 12957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41093
I20260620 14:21:28.658478 13086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41093 every 8 connection(s)
I20260620 14:21:28.658806 12957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:28.660843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 12957
I20260620 14:21:28.660955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:21:28.661140 392 external_mini_cluster.cc:1428] Setting key d07ba9867efe4dd8e445c8ef6a61822a
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:28.669487 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:28.659619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47513 (local address 127.0.98.62:39509)
0620 14:21:28.659692 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:28.659694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:28.659927 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:21:28.660972 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:28.660973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:28.660986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:28.661038 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:28.661803 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.662539 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.663598 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.663895 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.663978 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:28.666469 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:28.666471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:28.666471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:28.666480 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:28.668101 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:28.668520 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:28.668521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:28.668521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:28.668585 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:28.668826 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:28.668827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:28.668827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:28.668960 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:28.669027 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:28.669300 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:21:28.669388 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:21:28.669595 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:28.659602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39509 (local address 127.0.98.3:47513)
0620 14:21:28.659794 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:28.659831 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:28.660822 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:28.661112 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:28.661121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:28.661396 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:28.661669 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:28.661674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.662632 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.662633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:28.663459 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:28.663461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:28.664019 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:28.664020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:28.664147 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:28.664788 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:28.664796 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:28.666362 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:28.668190 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:28.668195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:28.668198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:28.668439 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:28.668652 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:28.668653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:28.668653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:28.668759 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:28.669026 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:28.669030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:28.669163 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:21:28.669248 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:21:28.669381 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260620 14:21:28.670619 13087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39509
I20260620 14:21:28.670739 13087 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:28.670954 13087 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:28.671406 12603 ts_manager.cc:194] Registered new tserver with Master: 3f3e736b55bf40369a2d722934a662f2 (127.0.98.3:41093)
I20260620 14:21:28.671873 12603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47513
I20260620 14:21:28.672797 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:21:28.673345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 12684
I20260620 14:21:28.678915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 12821
I20260620 14:21:28.686967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 12957
I20260620 14:21:28.695490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 12571
I20260620 14:21:28.702821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39509
--webserver_interface=127.0.98.62
--webserver_port=43295
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39509
--ranger_config_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:28.785852 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:28.786082 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:28.786106 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:28.787401 13090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:28.787456 13090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:28.787472 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:28.787484 13090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:28.787496 13090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:28.789347 13090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39509
--ranger_config_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:28.789644 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:28.789920 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:21:28.795117 13090 server_base.cc:1034] running on GCE node
W20260620 14:21:28.795099 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.795212 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:28.795117 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:28.795673 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:28.795989 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:28.797219 13090 hybrid_clock.cc:648] HybridClock initialized: now 1781965288797213 us; error 41 us; skew 500 ppm
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:28.799948 13090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:28.800442 13090 webserver.cc:460] Webserver started at http://127.0.98.62:43295/ using document root <none> and password file <none>
I20260620 14:21:28.800717 13090 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:28.800803 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:21:28 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:28.843356 13090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:28.844272 13106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:28.844489 13090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:21:28.844604 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "c1941a9f3fee436791e5b0bab51be078"
format_stamp: "Formatted at 2026-06-20 14:21:27 on dist-test-slave-h6n5"
server_key: "b59799781a33e9b54e9c7b678446298c"
server_key_iv: "f414fd5df116db14c7d9e8f0c9251357"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:28.845032 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:28.878062 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:28.880061 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:28.880240 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:28.884536 13090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39509
I20260620 14:21:28.884567 13158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39509 every 8 connection(s)
I20260620 14:21:28.884966 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:28.886264 13159 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:21:28.886845 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Bootstrap starting.
I20260620 14:21:28.889901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13090
I20260620 14:21:28.890293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44917
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39509
--webserver_port=42113
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:28.895138 13159 log.cc:824] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:28.896993 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:21:28.897300 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Bootstrap complete.
I20260620 14:21:28.899822 13159 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:28.900106 13159 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1941a9f3fee436791e5b0bab51be078, State: Initialized, Role: FOLLOWER
I20260620 14:21:28.900321 13159 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:28.900482 13159 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:28.900542 13159 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:28.900584 13159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:21:28.902156 13159 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:28.902285 13159 leader_election.cc:302] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1941a9f3fee436791e5b0bab51be078; no voters:
I20260620 14:21:28.902447 13159 leader_election.cc:288] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:21:28.902714 13159 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:28.902830 13164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:21:28.903131 13164 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [term 2 LEADER]: Becoming Leader. State: Replica: c1941a9f3fee436791e5b0bab51be078, State: Running, Role: LEADER
I20260620 14:21:28.903399 13164 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } }
I20260620 14:21:28.904045 13164 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c1941a9f3fee436791e5b0bab51be078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } } }
I20260620 14:21:28.904240 13164 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:28.904405 13159 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:21:28.904623 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1941a9f3fee436791e5b0bab51be078. Latest consensus state: current_term: 2 leader_uuid: "c1941a9f3fee436791e5b0bab51be078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1941a9f3fee436791e5b0bab51be078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39509 } } }
I20260620 14:21:28.904717 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:28.907605 13178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:28.908586 13178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:28.908829 13178 catalog_manager.cc:1259] Loaded cluster ID: cd24bfcc9ba3413d8dadd0238b2ee868
I20260620 14:21:28.908897 13178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:28.910236 13178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:28.910501 13178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Loaded TSK: 0
I20260620 14:21:28.910753 13178 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:21:29.005172 13161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:29.005482 13161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:29.005522 13161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:29.007325 13161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:29.007388 13161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:29.007431 13161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:29.009605 13161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42113
--tserver_master_addrs=127.0.98.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:29.009877 13161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:29.010142 13161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:29.016530 13199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:29.016631 13197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:29.016544 13196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:29.018311 13161 server_base.cc:1034] running on GCE node
I20260620 14:21:29.018656 13161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:29.018950 13161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:29.020134 13161 hybrid_clock.cc:648] HybridClock initialized: now 1781965289020110 us; error 36 us; skew 500 ppm
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:29.023281 13161 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:29.023819 13161 webserver.cc:460] Webserver started at http://127.0.98.1:42113/ using document root <none> and password file <none>
I20260620 14:21:29.024084 13161 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:29.024173 13161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:29.052065 13161 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260620 14:21:29.052899 13207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.053092 13161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:29.053179 13161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "140110b0397c4d8297777d947426eb1c"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "905c44c52a740cde341b4122a9828f2d"
server_key_iv: "a9797d0d8a78a84aa51c25aacee372ec"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:29.053570 13161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:29.063478 13161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:29.065729 13161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:29.065905 13161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:29.066192 13161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:29.066596 13161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:29.066653 13161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.066694 13161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:29.066718 13161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.073278 13161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44917
I20260620 14:21:29.073515 13320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44917 every 8 connection(s)
I20260620 14:21:29.073832 13161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:29.079007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13161
I20260620 14:21:29.079329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43745
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39509
--webserver_port=37035
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:29.105986 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:29.082432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39509 (local address 127.0.98.1:44845)
0620 14:21:29.083023 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:29.083077 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:29.083952 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:29.084601 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:29.084605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:29.084877 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:29.085138 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:29.085143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.086831 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.086832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:29.088139 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:29.088142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.090584 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.090585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:29.090746 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:29.099744 (+ 8998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:29.099759 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:29.101708 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:29.104090 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:29.104099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:29.104101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:29.104400 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:29.104652 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:29.104652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:29.104653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:29.104838 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:29.105322 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:29.105328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:29.105484 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:21:29.105573 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:21:29.105747 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":89,"threads_started":1}
I20260620 14:21:29.106997 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:29.082816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44845 (local address 127.0.98.62:39509)
0620 14:21:29.083575 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:29.083585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:29.083630 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:21:29.084265 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:29.084271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:29.084331 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:29.084456 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:29.085258 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.086713 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.088237 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.088850 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.088951 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:29.101861 (+ 12910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:29.101867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:29.101868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:29.101889 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:29.103981 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.104484 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.104488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.104488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.104591 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.104906 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.104907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.104907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.105218 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:29.105294 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:29.106675 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0620 14:21:29.106802 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":72,"threads_started":1}
I20260620 14:21:29.107739 13321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39509
I20260620 14:21:29.107839 13321 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.108065 13321 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
W20260620 14:21:29.109033 13321 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:21:29.109094 13321 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.109149 13321 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:29.109500 13321 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.109565 13321 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
W20260620 14:21:29.109860 13321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:21:29.204857 13323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:29.205152 13323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:29.205188 13323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:29.206996 13323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:29.207058 13323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:29.207098 13323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:29.209299 13323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37035
--tserver_master_addrs=127.0.98.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:29.209553 13323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:29.209825 13323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:29.216457 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:29.216512 13331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:29.216614 13330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:29.217418 13323 server_base.cc:1034] running on GCE node
I20260620 14:21:29.217651 13323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:29.217937 13323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:29.219138 13323 hybrid_clock.cc:648] HybridClock initialized: now 1781965289219120 us; error 37 us; skew 500 ppm
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:29.222064 13323 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:29.222712 13323 webserver.cc:460] Webserver started at http://127.0.98.2:37035/ using document root <none> and password file <none>
I20260620 14:21:29.222981 13323 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:29.223071 13323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:29.249017 13323 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:21:29.249845 13341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.250026 13323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:29.250121 13323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "716025670b3844f0a8bf5ff3f46db52e"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "cd6d89ed3f4a41282b13659d2a352476"
server_key_iv: "06d598b5959ed95f329c31c8640e0fa7"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:29.250524 13323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:29.271806 13323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:29.274199 13323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:29.274407 13323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:29.274680 13323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:29.275027 13323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:29.275090 13323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.275134 13323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:29.275161 13323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.281831 13323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43745
I20260620 14:21:29.282239 13323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:29.283402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13323
I20260620 14:21:29.283782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41093
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39509
--webserver_port=32781
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:29.291668 13454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43745 every 8 connection(s)
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:29.296250 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:29.282988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34769 (local address 127.0.98.62:39509)
0620 14:21:29.283042 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:29.283044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:29.283289 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:21:29.284141 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:29.284142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:29.284159 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:29.284259 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:29.285013 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.285830 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.287427 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.287796 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.287860 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:29.292491 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:29.292493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:29.292494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:29.292504 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:29.294424 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.295003 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.295004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.295005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.295073 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.295389 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.295391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.295391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.295544 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:29.295641 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:29.296049 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 14:21:29.296147 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:21:29.296463 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:29.283004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39509 (local address 127.0.98.2:34769)
0620 14:21:29.283158 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:29.283200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:29.283986 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:29.284333 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:29.284338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:29.284597 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:29.284892 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:29.284897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.285968 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.285969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:29.287322 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:29.287325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.287973 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.287974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:29.288134 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:29.288790 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:29.288801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:29.292345 (+ 3544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:29.294523 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:29.294531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:29.294534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:29.294898 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:29.295159 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:29.295160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:29.295161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:29.295304 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:29.295675 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:29.295681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:29.295875 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 14:21:29.296021 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:21:29.296193 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260620 14:21:29.296986 13455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39509
I20260620 14:21:29.297096 13455 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.297304 13455 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
W20260620 14:21:29.298102 13455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:21:29.298183 13455 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.298277 13455 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:29.298622 13455 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.298713 13455 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
W20260620 14:21:29.299052 13455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:21:29.398640 13458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:29.398917 13458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:29.398962 13458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:29.400768 13458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:29.400825 13458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:29.400868 13458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:29.403098 13458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32781
--tserver_master_addrs=127.0.98.62:39509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:29.403402 13458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:29.403698 13458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:29.410161 13464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:29.410249 13467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:29.410477 13465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:29.410646 13458 server_base.cc:1034] running on GCE node
I20260620 14:21:29.410851 13458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:29.411181 13458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:29.412352 13458 hybrid_clock.cc:648] HybridClock initialized: now 1781965289412342 us; error 36 us; skew 500 ppm
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:29.415673 13458 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:29.416249 13458 webserver.cc:460] Webserver started at http://127.0.98.3:32781/ using document root <none> and password file <none>
I20260620 14:21:29.416538 13458 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:29.416627 13458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:29.430652 13458 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:29.431504 13475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.431768 13458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:29.431892 13458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "3f3e736b55bf40369a2d722934a662f2"
format_stamp: "Formatted at 2026-06-20 14:21:28 on dist-test-slave-h6n5"
server_key: "fa5183ac54d467f2ce6fe2c5404ba800"
server_key_iv: "37c5d549dd10252e05162070128fb40a"
server_key_version: "kuduclusterkey@0"
I20260620 14:21:29.432291 13458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:29.455415 13458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:29.457767 13458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:29.458002 13458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:29.458298 13458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:29.458688 13458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:29.458746 13458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.458787 13458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:29.458815 13458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:29.465617 13458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41093
I20260620 14:21:29.465922 13588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41093 every 8 connection(s)
I20260620 14:21:29.466116 13458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:29.474659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13458
I20260620 14:21:29.477701 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:29.467591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39509 (local address 127.0.98.3:51701)
0620 14:21:29.467745 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:29.467787 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:29.468411 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:29.468689 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:29.468693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:29.468899 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:29.469165 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:29.469170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.469916 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.469917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:29.470733 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:29.470735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.471190 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.471191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:29.471322 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:29.471990 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:29.472001 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:29.473772 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:29.475933 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:29.475941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:29.475944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:29.476260 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:29.476506 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:29.476507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:29.476508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:29.476659 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:29.476997 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:29.477004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:29.477198 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 14:21:29.477294 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:21:29.477456 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260620 14:21:29.478091 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:29.467570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51701 (local address 127.0.98.62:39509)
0620 14:21:29.467622 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:29.467624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:29.467925 (+ 301us) server_negotiation.cc:407] Connection header received
0620 14:21:29.468531 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:29.468533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:29.468546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:29.468604 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:29.469269 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.469780 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.470841 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.471066 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.471165 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:29.473909 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:29.473911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:29.473912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:29.473920 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:29.475835 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.476352 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.476354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.476354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.476421 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.476743 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.476744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.476744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.476910 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:29.477871 (+ 961us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:29.477922 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:21:29.478010 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:21:29.478554 13589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39509
I20260620 14:21:29.478664 13589 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.478876 13589 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
W20260620 14:21:29.479640 13589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:21:29.479722 13589 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.479794 13589 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:29.480090 13589 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:29.480167 13589 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
W20260620 14:21:29.480448 13589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:21:29.484596 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:29.475992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.0.98.62:39509)
0620 14:21:29.476416 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:29.476418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:29.476425 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:21:29.476444 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:29.476444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:29.476456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:29.477272 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:29.477310 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.478548 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.479682 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:29.480095 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:29.480784 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:29.481689 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:29.481691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:29.481691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:29.481698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:29.483255 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.483705 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.483706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.483707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.483797 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:29.484028 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:29.484029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:29.484029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:29.484146 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:29.484348 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:29.484392 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:21:29.484479 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":65,"threads_started":1}
Jun 20 14:21:29 dist-test-slave-h6n5 krb5kdc[12127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:30.110036 13321 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:30.110193 13321 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:30.110598 13122 ts_manager.cc:194] Registered new tserver with Master: 140110b0397c4d8297777d947426eb1c (127.0.98.1:44917)
I20260620 14:21:30.111655 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44845
I20260620 14:21:30.299237 13455 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:30.299417 13455 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:30.299877 13122 ts_manager.cc:194] Registered new tserver with Master: 716025670b3844f0a8bf5ff3f46db52e (127.0.98.2:43745)
I20260620 14:21:30.300410 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34769
Jun 20 14:21:30 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:21:30.480641 13589 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:30.480819 13589 heartbeater.cc:511] Master 127.0.98.62:39509 requested a full tablet report, sending...
I20260620 14:21:30.481101 13122 ts_manager.cc:194] Registered new tserver with Master: 3f3e736b55bf40369a2d722934a662f2 (127.0.98.3:41093)
I20260620 14:21:30.481652 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51701
I20260620 14:21:30.492174 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:21:30.503849 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.493206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.0.98.62:39509)
0620 14:21:30.495683 (+ 2477us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:30.495686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:30.495815 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:21:30.495868 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:30.495869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:30.495882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:30.495954 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:30.496672 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.497481 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.498565 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.498878 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.498945 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.500569 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:30.500571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:30.500572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:30.500580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:30.502484 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:30.502888 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:30.502890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:30.502890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:30.502955 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:30.503192 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:30.503193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:30.503194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:30.503333 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:30.503437 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:30.503651 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:21:30.503741 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2454,"thread_start_us":68,"threads_started":1}
I20260620 14:21:30.506250 13122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:21:30.508940 13122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:21:30.517306 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.511915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43745 (local address 127.0.0.1:57528)
0620 14:21:30.512983 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:30.512991 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:30.513031 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:30.513264 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:30.513265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:30.513267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:30.513311 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.513312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.514538 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.514539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.516001 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.516004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.517018 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.517019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.517098 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.517111 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:21:30.517175 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:21:30.517224 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260620 14:21:30.517573 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.511913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41093 (local address 127.0.0.1:42996)
0620 14:21:30.512649 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:30.512688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:30.512798 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:30.513102 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:30.513105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:30.513108 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:30.513183 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.513185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.514740 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.514741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.515999 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.516002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.517335 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.517336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.517398 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.517402 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:30.517455 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:21:30.517494 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260620 14:21:30.517937 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.512165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.0.98.3:41093)
0620 14:21:30.512299 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:30.512304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:30.512756 (+ 452us) server_negotiation.cc:407] Connection header received
0620 14:21:30.512897 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:30.512900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:30.512918 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:30.513005 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:30.513285 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.514657 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.516098 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.517144 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.517213 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.517299 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:30.517823 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 14:21:30.517864 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260620 14:21:30.518782 13523 tablet_service.cc:1467] Processing CreateTablet for tablet f273ca889acf4ec9a3b62adcb1c8e383 (DEFAULT_TABLE table=test-table [id=6b4a389b0fec46ba92f3e6df2119d491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:30.519121 13523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f273ca889acf4ec9a3b62adcb1c8e383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:30.520634 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.512379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.0.98.2:43745)
0620 14:21:30.512810 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:30.512815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:30.513082 (+ 267us) server_negotiation.cc:407] Connection header received
0620 14:21:30.513118 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:30.513121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:30.513134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:30.513196 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:30.513391 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.514444 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.516099 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.516835 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.516908 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.516982 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:30.520508 (+ 3526us) server_negotiation.cc:299] Negotiation successful
0620 14:21:30.520556 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":57,"threads_started":1}
I20260620 14:21:30.521584 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.512122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44917 (local address 127.0.0.1:48190)
0620 14:21:30.517732 (+ 5610us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:30.517741 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:30.517807 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:30.518033 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:30.518034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:30.518035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:30.518063 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.518064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.519296 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.519296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.520396 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.520399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.521357 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.521358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.521421 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.521424 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:21:30.521483 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:21:30.521520 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5599,"thread_start_us":51,"threads_started":1}
I20260620 14:21:30.521610 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.512512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.98.1:44917)
0620 14:21:30.512938 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:30.512942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:30.517781 (+ 4839us) server_negotiation.cc:407] Connection header received
0620 14:21:30.517885 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:30.517888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:30.517903 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:30.517952 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:30.518128 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.519200 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.520501 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.521245 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.521330 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.521399 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:30.521494 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:21:30.521538 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":47,"threads_started":1}
I20260620 14:21:30.522514 13622 tablet_bootstrap.cc:492] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Bootstrap starting.
I20260620 14:21:30.522553 13255 tablet_service.cc:1467] Processing CreateTablet for tablet f273ca889acf4ec9a3b62adcb1c8e383 (DEFAULT_TABLE table=test-table [id=6b4a389b0fec46ba92f3e6df2119d491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:30.522871 13255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f273ca889acf4ec9a3b62adcb1c8e383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:30.523402 13389 tablet_service.cc:1467] Processing CreateTablet for tablet f273ca889acf4ec9a3b62adcb1c8e383 (DEFAULT_TABLE table=test-table [id=6b4a389b0fec46ba92f3e6df2119d491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:30.523726 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f273ca889acf4ec9a3b62adcb1c8e383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:30.523797 13622 tablet_bootstrap.cc:654] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:30.524066 13622 log.cc:824] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:30.526140 13624 tablet_bootstrap.cc:492] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c: Bootstrap starting.
I20260620 14:21:30.526829 13624 tablet_bootstrap.cc:654] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:30.527022 13622 tablet_bootstrap.cc:492] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: No bootstrap required, opened a new log
I20260620 14:21:30.527208 13624 log.cc:824] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:30.527238 13622 ts_tablet_manager.cc:1397] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:21:30.528019 13624 tablet_bootstrap.cc:492] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c: No bootstrap required, opened a new log
I20260620 14:21:30.528080 13624 ts_tablet_manager.cc:1397] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:30.529737 13622 raft_consensus.cc:348] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.529896 13622 raft_consensus.cc:374] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:30.529927 13622 raft_consensus.cc:729] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3e736b55bf40369a2d722934a662f2, State: Initialized, Role: FOLLOWER
I20260620 14:21:30.530061 13622 consensus_queue.cc:260] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.530301 13622 ts_tablet_manager.cc:1428] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:30.530419 13589 heartbeater.cc:503] Master 127.0.98.62:39509 was elected leader, sending a full tablet report...
I20260620 14:21:30.530612 13624 raft_consensus.cc:348] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.530761 13624 raft_consensus.cc:374] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:30.530799 13624 raft_consensus.cc:729] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140110b0397c4d8297777d947426eb1c, State: Initialized, Role: FOLLOWER
I20260620 14:21:30.530887 13625 tablet_bootstrap.cc:492] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Bootstrap starting.
I20260620 14:21:30.531172 13624 consensus_queue.cc:260] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.531457 13624 ts_tablet_manager.cc:1428] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:30.531528 13625 tablet_bootstrap.cc:654] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:30.531601 13321 heartbeater.cc:503] Master 127.0.98.62:39509 was elected leader, sending a full tablet report...
I20260620 14:21:30.532287 13625 log.cc:824] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:30.533478 13625 tablet_bootstrap.cc:492] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: No bootstrap required, opened a new log
I20260620 14:21:30.533555 13625 ts_tablet_manager.cc:1397] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:21:30.536616 13625 raft_consensus.cc:348] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.536799 13625 raft_consensus.cc:374] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:30.536840 13625 raft_consensus.cc:729] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716025670b3844f0a8bf5ff3f46db52e, State: Initialized, Role: FOLLOWER
I20260620 14:21:30.537003 13625 consensus_queue.cc:260] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.537271 13625 ts_tablet_manager.cc:1428] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:21:30.537298 13455 heartbeater.cc:503] Master 127.0.98.62:39509 was elected leader, sending a full tablet report...
W20260620 14:21:30.548929 13456 tablet.cc:2369] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:21:30.582967 13322 tablet.cc:2369] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:21:30.587899 13627 raft_consensus.cc:482] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:30.588111 13627 raft_consensus.cc:504] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.588616 13627 leader_election.cc:288] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716025670b3844f0a8bf5ff3f46db52e (127.0.98.2:43745), 140110b0397c4d8297777d947426eb1c (127.0.98.1:44917)
I20260620 14:21:30.592628 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.588864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33607 (local address 127.0.98.2:43745)
0620 14:21:30.588913 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:30.588915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:30.589058 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:21:30.589167 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:30.589168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:30.589182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:30.589211 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:30.589452 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.590188 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.591626 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.592229 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.592279 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.592290 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:30.592518 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:21:30.592560 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:21:30.593211 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.588796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43745 (local address 127.0.98.3:33607)
0620 14:21:30.588991 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:30.589002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:30.589086 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:30.589283 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:30.589284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:30.589286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:30.589379 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.589381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.590296 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.590296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.591478 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.591480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.592363 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.592364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.592433 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.592436 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:21:30.593066 (+ 630us) client_negotiation.cc:240] Negotiation successful
0620 14:21:30.593111 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260620 14:21:30.593768 13409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273ca889acf4ec9a3b62adcb1c8e383" candidate_uuid: "3f3e736b55bf40369a2d722934a662f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716025670b3844f0a8bf5ff3f46db52e" is_pre_election: true
I20260620 14:21:30.593928 13409 raft_consensus.cc:2454] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3e736b55bf40369a2d722934a662f2 in term 0.
I20260620 14:21:30.594231 13478 leader_election.cc:302] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3e736b55bf40369a2d722934a662f2, 716025670b3844f0a8bf5ff3f46db52e; no voters:
I20260620 14:21:30.594362 13627 raft_consensus.cc:2781] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:21:30.594401 13627 raft_consensus.cc:482] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:21:30.594424 13627 raft_consensus.cc:3037] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:30.595788 13627 raft_consensus.cc:504] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.595911 13627 leader_election.cc:288] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [CANDIDATE]: Term 1 election: Requested vote from peers 716025670b3844f0a8bf5ff3f46db52e (127.0.98.2:43745), 140110b0397c4d8297777d947426eb1c (127.0.98.1:44917)
I20260620 14:21:30.596205 13409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273ca889acf4ec9a3b62adcb1c8e383" candidate_uuid: "3f3e736b55bf40369a2d722934a662f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716025670b3844f0a8bf5ff3f46db52e"
I20260620 14:21:30.596280 13409 raft_consensus.cc:3037] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:30.597133 13409 raft_consensus.cc:2454] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3e736b55bf40369a2d722934a662f2 in term 1.
I20260620 14:21:30.597359 13478 leader_election.cc:302] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3e736b55bf40369a2d722934a662f2, 716025670b3844f0a8bf5ff3f46db52e; no voters:
I20260620 14:21:30.597489 13627 raft_consensus.cc:2781] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:30.597643 13627 raft_consensus.cc:686] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3e736b55bf40369a2d722934a662f2, State: Running, Role: LEADER
I20260620 14:21:30.597738 13627 consensus_queue.cc:237] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } }
I20260620 14:21:30.598563 13122 catalog_manager.cc:5495] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3e736b55bf40369a2d722934a662f2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3f3e736b55bf40369a2d722934a662f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3e736b55bf40369a2d722934a662f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 } health_report { overall_health: UNKNOWN } } }
I20260620 14:21:30.602672 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.589140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51261 (local address 127.0.98.1:44917)
0620 14:21:30.589187 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:30.589189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:30.589302 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:21:30.589399 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:30.589400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:30.589414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:30.589443 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:30.589664 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.590390 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.598443 (+ 8053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.599027 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.599080 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.599092 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:30.602537 (+ 3445us) server_negotiation.cc:299] Negotiation successful
0620 14:21:30.602597 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:21:30.603047 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.588991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44917 (local address 127.0.98.3:51261)
0620 14:21:30.589246 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:30.589255 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:30.589327 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:30.589506 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:30.589507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:30.589508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:30.589597 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.589598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.597213 (+ 7615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.597214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.598326 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:30.598329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.602363 (+ 4034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.602364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:30.602442 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.602446 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:30.602919 (+ 473us) client_negotiation.cc:240] Negotiation successful
0620 14:21:30.602965 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":51,"threads_started":1}
I20260620 14:21:30.603608 13275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273ca889acf4ec9a3b62adcb1c8e383" candidate_uuid: "3f3e736b55bf40369a2d722934a662f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140110b0397c4d8297777d947426eb1c" is_pre_election: true
I20260620 14:21:30.603765 13275 raft_consensus.cc:2454] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3e736b55bf40369a2d722934a662f2 in term 0.
I20260620 14:21:30.603895 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f273ca889acf4ec9a3b62adcb1c8e383" candidate_uuid: "3f3e736b55bf40369a2d722934a662f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140110b0397c4d8297777d947426eb1c"
I20260620 14:21:30.603950 13274 raft_consensus.cc:3037] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:30.604882 13274 raft_consensus.cc:2454] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3e736b55bf40369a2d722934a662f2 in term 1.
I20260620 14:21:30.611905 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.608697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.0.98.3:41093)
0620 14:21:30.608763 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:30.608765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:30.608773 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:21:30.608807 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:30.608808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:30.608828 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:30.608906 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:21:30.609094 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.609798 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.610917 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:30.611213 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:30.611294 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:30.611504 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:21:30.611654 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:21:30.611734 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:30.611816 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:21:30.611835 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:21:30.613759 13274 raft_consensus.cc:1261] T f273ca889acf4ec9a3b62adcb1c8e383 P 140110b0397c4d8297777d947426eb1c [term 1 FOLLOWER]: Refusing update from remote peer 3f3e736b55bf40369a2d722934a662f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:30.613840 13409 raft_consensus.cc:1261] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 1 FOLLOWER]: Refusing update from remote peer 3f3e736b55bf40369a2d722934a662f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:30.614181 13633 consensus_queue.cc:1035] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "716025670b3844f0a8bf5ff3f46db52e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:30.614312 13627 consensus_queue.cc:1035] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140110b0397c4d8297777d947426eb1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:30.618114 13637 mvcc.cc:205] Tried to move back new op lower bound from 7298929830351200256 to 7298929830288912384. Current Snapshot: MvccSnapshot[applied={T|T < 7298929830351200256 or (T in {7298929830351200256})}]
I20260620 14:21:30.619091 13123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:30.619180 13123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:30.620029 13123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1941a9f3fee436791e5b0bab51be078: Sending DeleteTablet for 3 replicas of tablet f273ca889acf4ec9a3b62adcb1c8e383
I20260620 14:21:30.620451 13389 tablet_service.cc:1514] Processing DeleteTablet for tablet f273ca889acf4ec9a3b62adcb1c8e383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57528
I20260620 14:21:30.620659 13255 tablet_service.cc:1514] Processing DeleteTablet for tablet f273ca889acf4ec9a3b62adcb1c8e383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48190
I20260620 14:21:30.621055 13523 tablet_service.cc:1514] Processing DeleteTablet for tablet f273ca889acf4ec9a3b62adcb1c8e383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42996
I20260620 14:21:30.621681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13161
I20260620 14:21:30.624001 13638 mvcc.cc:205] Tried to move back new op lower bound from 7298929830351200256 to 7298929830288912384. Current Snapshot: MvccSnapshot[applied={T|T < 7298929830351200256}]
I20260620 14:21:30.624466 13646 tablet_replica.cc:330] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: stopping tablet replica
I20260620 14:21:30.624650 13646 raft_consensus.cc:2229] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:21:30.624795 13646 raft_consensus.cc:2258] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:30.624976 13644 tablet_replica.cc:330] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: stopping tablet replica
I20260620 14:21:30.631881 13646 ts_tablet_manager.cc:1905] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:30.632095 13644 raft_consensus.cc:2229] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:30.632246 13644 raft_consensus.cc:2258] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:30.635658 13646 ts_tablet_manager.cc:1918] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:30.635749 13646 log.cc:1196] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/f273ca889acf4ec9a3b62adcb1c8e383
I20260620 14:21:30.635885 13644 ts_tablet_manager.cc:1905] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:30.636070 13646 ts_tablet_manager.cc:1939] T f273ca889acf4ec9a3b62adcb1c8e383 P 3f3e736b55bf40369a2d722934a662f2: Deleting consensus metadata
W20260620 14:21:30.637713 13478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44917 (error 108)
I20260620 14:21:30.638224 13644 ts_tablet_manager.cc:1918] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:30.638432 13644 log.cc:1196] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/f273ca889acf4ec9a3b62adcb1c8e383
I20260620 14:21:30.638743 13644 ts_tablet_manager.cc:1939] T f273ca889acf4ec9a3b62adcb1c8e383 P 716025670b3844f0a8bf5ff3f46db52e: Deleting consensus metadata
I20260620 14:21:30.639232 13109 catalog_manager.cc:4841] TS 716025670b3844f0a8bf5ff3f46db52e (127.0.98.2:43745): tablet f273ca889acf4ec9a3b62adcb1c8e383 (table test-table [id=6b4a389b0fec46ba92f3e6df2119d491]) successfully deleted
W20260620 14:21:30.639494 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44917 (error 108)
W20260620 14:21:30.640107 13616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:30.639912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44917 (local address 127.0.0.1:48202)
0620 14:21:30.640001 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:30.640056 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44917: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":57}
Jun 20 14:21:30 dist-test-slave-h6n5 krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 14:21:30.640285 13109 catalog_manager.cc:4584] TS 140110b0397c4d8297777d947426eb1c (127.0.98.1:44917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f273ca889acf4ec9a3b62adcb1c8e383: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44917: connect: Connection reset by peer (error 104)
I20260620 14:21:30.640720 13107 catalog_manager.cc:4841] TS 3f3e736b55bf40369a2d722934a662f2 (127.0.98.3:41093): tablet f273ca889acf4ec9a3b62adcb1c8e383 (table test-table [id=6b4a389b0fec46ba92f3e6df2119d491]) successfully deleted
I20260620 14:21:30.640815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13323
I20260620 14:21:30.640892 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:30.640633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44917 (local address 127.0.98.3:33831)
0620 14:21:30.640823 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:30.640864 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260620 14:21:30.647315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13458
I20260620 14:21:30.655624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13090
I20260620 14:21:30.664594 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:21:31.009799 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:21:31.009909 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:21:31.354658 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:21:31.354 UTC [12157] LOG: received smart shutdown request
2026-06-20 14:21:31.358 UTC [12157] LOG: background worker "logical replication launcher" (PID 12166) exited with exit code 1
2026-06-20 14:21:31.359 UTC [12161] LOG: shutting down
2026-06-20 14:21:31.553 UTC [12157] LOG: database system is shut down
2026-06-20T14:21:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48454 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:21:31 dist-test-slave-h6n5 krb5kdc[13659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:21:31 dist-test-slave-h6n5 krb5kdc[13659](info): set up 2 sockets
Jun 20 14:21:31 dist-test-slave-h6n5 krb5kdc[13659](info): commencing operation
krb5kdc: starting...
W20260620 14:21:32.703879 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:21:32 dist-test-slave-h6n5 krb5kdc[13659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:21:32.822707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869,127.0.98.61:38091,127.0.98.60:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:32.904882 13672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:32.905103 13672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:32.905125 13672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:32.906219 13672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:32.906255 13672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:32.906270 13672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:32.906282 13672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:32.906294 13672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:32.907893 13672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869,127.0.98.61:38091,127.0.98.60:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:32.908087 13672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:32.908262 13672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:32.913503 13680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:32.913512 13678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:32.913802 13672 server_base.cc:1034] running on GCE node
W20260620 14:21:32.913553 13677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:32.914095 13672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:32.914431 13672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:32.915643 13672 hybrid_clock.cc:648] HybridClock initialized: now 1781965292915628 us; error 34 us; skew 500 ppm
Jun 20 14:21:32 dist-test-slave-h6n5 krb5kdc[13659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:32.922049 13672 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:32.922565 13672 webserver.cc:460] Webserver started at http://127.0.98.62:36219/ using document root <none> and password file <none>
I20260620 14:21:32.922864 13672 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:32.922947 13672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:32.923116 13672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:32.924396 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "5222b10fe70e4b60b9a6952cc61c38d3"
format_stamp: "Formatted at 2026-06-20 14:21:32 on dist-test-slave-h6n5"
server_key: "7c73fd9968f93733a48586036d90d024"
server_key_iv: "04e9648c09ba184f8ac7a4a71ba4c9be"
server_key_version: "encryptionkey@0"
I20260620 14:21:32.924815 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "5222b10fe70e4b60b9a6952cc61c38d3"
format_stamp: "Formatted at 2026-06-20 14:21:32 on dist-test-slave-h6n5"
server_key: "7c73fd9968f93733a48586036d90d024"
server_key_iv: "04e9648c09ba184f8ac7a4a71ba4c9be"
server_key_version: "encryptionkey@0"
I20260620 14:21:32.926343 13672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:21:32.927369 13687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:32.927580 13672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:32.927678 13672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "5222b10fe70e4b60b9a6952cc61c38d3"
format_stamp: "Formatted at 2026-06-20 14:21:32 on dist-test-slave-h6n5"
server_key: "7c73fd9968f93733a48586036d90d024"
server_key_iv: "04e9648c09ba184f8ac7a4a71ba4c9be"
server_key_version: "encryptionkey@0"
I20260620 14:21:32.927772 13672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:32.944278 13672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:32.946530 13672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:32.946736 13672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:32.950660 13672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44869
I20260620 14:21:32.950695 13739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44869 every 8 connection(s)
I20260620 14:21:32.951083 13672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:32.952122 13740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:32.953745 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:21:32.957767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13672
I20260620 14:21:32.957861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:32.958113 392 external_mini_cluster.cc:1428] Setting key 5659d7b342d31d198eafac2947bafa0e
Jun 20 14:21:32 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:32.964486 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:32.954537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.0.98.62:44869)
0620 14:21:32.954695 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:32.954703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:32.954797 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:21:32.955344 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:32.955346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:32.955363 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:32.955459 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:32.956101 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:32.956937 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:32.957954 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:32.958473 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:32.958588 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:32.961031 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:32.961038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:32.961038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:32.961045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:32.962706 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:32.963156 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:32.963160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:32.963160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:32.963242 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:32.963488 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:32.963489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:32.963489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:32.963756 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:32.963856 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:32.964047 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:21:32.964236 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":118,"thread_start_us":96,"threads_started":1}
I20260620 14:21:32.964488 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:32.954485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.0.1:57372)
0620 14:21:32.954666 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:32.954704 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:32.955183 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:32.955484 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:32.955487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:32.955814 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:32.956006 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:32.956011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:32.957091 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:32.957092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:32.957790 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:32.957792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:32.958566 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:32.958567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:32.958631 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:32.959200 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:32.959208 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:32.960916 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:32.962849 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:32.962855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:32.962858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:32.963054 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:32.963329 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:32.963331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:32.963331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:32.963416 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:32.963831 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:32.963835 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:32.963897 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 14:21:32.964020 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:21:32.964284 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":99,"thread_start_us":94,"threads_started":1}
I20260620 14:21:32.965579 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 14:21:32.965987 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:32.965861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44212)
0620 14:21:32.965913 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:32.965963 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:21:32.966147 13690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
I20260620 14:21:32.966585 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:32.966507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44214)
0620 14:21:32.966545 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:32.966569 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:21:32.966718 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38091: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:21:32.982897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38091
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869,127.0.98.61:38091,127.0.98.60:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:33.015964 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } attempt: 1
I20260620 14:21:33.016587 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.016423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44228)
0620 14:21:33.016508 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.016557 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:21:33.017023 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.016894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44240)
0620 14:21:33.016965 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.016997 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:21:33.017159 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38091: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
W20260620 14:21:33.063688 13747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:33.063970 13747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:33.064007 13747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:33.065171 13747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:33.065254 13747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:33.065279 13747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:33.065299 13747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:33.065320 13747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:33.066854 13747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869,127.0.98.61:38091,127.0.98.60:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:33.067183 13747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:33.067404 13747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:33.072161 13752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:33.072181 13755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:33.072161 13753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:33.072394 13747 server_base.cc:1034] running on GCE node
I20260620 14:21:33.072631 13747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:33.072913 13747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:33.074079 13747 hybrid_clock.cc:648] HybridClock initialized: now 1781965293074073 us; error 43 us; skew 500 ppm
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:33.076488 13747 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:21:33.076988 13747 webserver.cc:460] Webserver started at http://127.0.98.61:37635/ using document root <none> and password file <none>
I20260620 14:21:33.077208 13747 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:33.077279 13747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:33.077407 13747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:33.078459 13747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data/instance:
uuid: "f65562f92c344c38878e2a129276bb3a"
format_stamp: "Formatted at 2026-06-20 14:21:33 on dist-test-slave-h6n5"
server_key: "bead23d3ddb3b1e0b98f3d6086bd3437"
server_key_iv: "717736917a332101a8a3d85f02191faf"
server_key_version: "encryptionkey@0"
I20260620 14:21:33.078830 13747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/wal/instance:
uuid: "f65562f92c344c38878e2a129276bb3a"
format_stamp: "Formatted at 2026-06-20 14:21:33 on dist-test-slave-h6n5"
server_key: "bead23d3ddb3b1e0b98f3d6086bd3437"
server_key_iv: "717736917a332101a8a3d85f02191faf"
server_key_version: "encryptionkey@0"
I20260620 14:21:33.080214 13747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:21:33.081013 13762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:33.081199 13747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:33.081317 13747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/wal
uuid: "f65562f92c344c38878e2a129276bb3a"
format_stamp: "Formatted at 2026-06-20 14:21:33 on dist-test-slave-h6n5"
server_key: "bead23d3ddb3b1e0b98f3d6086bd3437"
server_key_iv: "717736917a332101a8a3d85f02191faf"
server_key_version: "encryptionkey@0"
I20260620 14:21:33.081411 13747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:33.085307 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } attempt: 2
I20260620 14:21:33.085700 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.085570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44244)
0620 14:21:33.085635 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.085673 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:21:33.086051 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.085944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44260)
0620 14:21:33.086001 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.086024 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:21:33.086138 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38091: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38091: connect: Connection refused (error 111)
I20260620 14:21:33.089699 13747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:33.091699 13747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:33.091862 13747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:33.095914 13747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38091
I20260620 14:21:33.095947 13814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38091 every 8 connection(s)
I20260620 14:21:33.096290 13747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:21:33.097108 13815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:33.097985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13747
I20260620 14:21:33.098106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-1/wal/instance
I20260620 14:21:33.098315 392 external_mini_cluster.cc:1428] Setting key 948709f9f7999bca93a5174aac971e1d
I20260620 14:21:33.099098 13815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 14:21:33.108891 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.099986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.0.1:57386)
0620 14:21:33.100160 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.100205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.100697 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.100961 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.100965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.101219 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.101409 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.101412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.102113 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.102114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.102909 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.102910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.103360 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.103362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.103497 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.104098 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.104126 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.105482 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.107385 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.107392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.107394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.107700 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.107948 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.107948 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.107949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.108078 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.108382 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.108388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.108510 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.108606 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.108727 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260620 14:21:33.108894 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.100025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.0.98.62:44869)
0620 14:21:33.100071 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.100073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.100297 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:21:33.100829 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.100831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.100843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.100886 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.101514 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.101999 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.103042 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.103258 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.103351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.105594 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.105596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.105596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.105604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.107284 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.107795 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.107796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.107797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.107856 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.108155 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.108156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.108156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.108301 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.108373 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.108655 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.108751 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:21:33.109555 13815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:21:33.117206 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.109839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44266)
0620 14:21:33.109896 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.109905 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.109940 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.110161 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.110162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.110322 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.110391 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.110392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.111218 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.111219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.111851 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.111853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.112529 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.112530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.112585 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.113018 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.113023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.114150 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.116183 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.116184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.116184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.116338 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.116561 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.116562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.116562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.116611 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.116968 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.116968 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.117011 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.117069 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.117143 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:21:33.117323 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.109909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.0.98.61:38091)
0620 14:21:33.110037 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.110041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.110055 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:21:33.110069 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.110071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.110080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.110128 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.110471 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.111144 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.111933 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.112451 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.112534 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.114236 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.114244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.114245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.114252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.116095 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.116426 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.116429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.116430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.116489 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.116676 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.116677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.116677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.116866 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.116957 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.117096 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.117195 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260620 14:21:33.117934 13815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:21:33.118286 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.118155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35694)
0620 14:21:33.118212 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.118263 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:21:33.118439 13766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
I20260620 14:21:33.118933 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.118821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35698)
0620 14:21:33.118875 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.118902 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:21:33.119037 13815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:21:33.126642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40023
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869,127.0.98.61:38091,127.0.98.60:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:33.168325 13815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } attempt: 1
I20260620 14:21:33.168911 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.168764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35712)
0620 14:21:33.168835 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.168879 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:21:33.169323 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.169174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35714)
0620 14:21:33.169249 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.169293 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:21:33.169472 13815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
I20260620 14:21:33.177305 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } attempt: 3
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:21:33.184895 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.177784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.98.61:38091)
0620 14:21:33.177860 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.177863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.177875 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:21:33.177965 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.177967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.177983 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.178057 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.178399 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.178910 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.179782 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.180001 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.180087 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.182193 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.182195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.182195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.182202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.183797 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.184200 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.184201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.184201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.184252 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.184453 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.184454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.184454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.184551 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.184595 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.184766 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.184829 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:21:33.184963 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.177702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44272)
0620 14:21:33.177773 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.177786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.177873 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.178062 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.178063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.178229 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.178294 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.178296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.179028 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.179030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.179680 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.179683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.180085 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.180086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.180173 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.180693 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.180699 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.182091 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.183902 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.183904 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.183904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.184127 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.184324 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.184325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.184325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.184393 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.184618 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.184619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.184677 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.184752 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.184863 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:21:33.185575 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:21:33.185951 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.185835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35730)
0620 14:21:33.185894 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.185925 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:21:33.186313 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.186214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35746)
0620 14:21:33.186275 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.186298 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:21:33.186467 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
W20260620 14:21:33.211946 13822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:33.212152 13822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:33.212175 13822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:33.213290 13822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:33.213325 13822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:33.213338 13822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:33.213351 13822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:33.213363 13822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:33.214818 13822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869,127.0.98.61:38091,127.0.98.60:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:33.215013 13822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:33.215190 13822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:21:33.217636 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } attempt: 1
I20260620 14:21:33.218122 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.217961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35760)
0620 14:21:33.218035 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.218084 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:21:33.218621 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.218475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35774)
0620 14:21:33.218563 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.218599 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 14:21:33.218761 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
W20260620 14:21:33.220306 13828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:33.220302 13827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:33.220328 13830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:33.220522 13822 server_base.cc:1034] running on GCE node
I20260620 14:21:33.220827 13822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:33.221194 13822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:33.222402 13822 hybrid_clock.cc:648] HybridClock initialized: now 1781965293222371 us; error 39 us; skew 500 ppm
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:33.225021 13822 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:21:33.225522 13822 webserver.cc:460] Webserver started at http://127.0.98.60:37667/ using document root <none> and password file <none>
I20260620 14:21:33.225778 13822 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:33.225852 13822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:33.226015 13822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:33.227087 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data/instance:
uuid: "54a72bc7797349e7926929ef786c3d71"
format_stamp: "Formatted at 2026-06-20 14:21:33 on dist-test-slave-h6n5"
server_key: "0230cf25bab2a01c9c856637e310575b"
server_key_iv: "310e70eceab6f6bf8154a2f0571b23ca"
server_key_version: "encryptionkey@0"
I20260620 14:21:33.227453 13822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/wal/instance:
uuid: "54a72bc7797349e7926929ef786c3d71"
format_stamp: "Formatted at 2026-06-20 14:21:33 on dist-test-slave-h6n5"
server_key: "0230cf25bab2a01c9c856637e310575b"
server_key_iv: "310e70eceab6f6bf8154a2f0571b23ca"
server_key_version: "encryptionkey@0"
I20260620 14:21:33.228809 13822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:33.229549 13837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:33.229725 13822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:21:33.229846 13822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/wal
uuid: "54a72bc7797349e7926929ef786c3d71"
format_stamp: "Formatted at 2026-06-20 14:21:33 on dist-test-slave-h6n5"
server_key: "0230cf25bab2a01c9c856637e310575b"
server_key_iv: "310e70eceab6f6bf8154a2f0571b23ca"
server_key_version: "encryptionkey@0"
I20260620 14:21:33.229929 13822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:33.237665 13815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } attempt: 2
I20260620 14:21:33.238147 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.237999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35784)
0620 14:21:33.238065 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.238109 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:21:33.238539 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.238455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35794)
0620 14:21:33.238498 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.238522 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:21:33.238667 13815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40023: connect: Connection refused (error 111)
I20260620 14:21:33.241374 13822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:33.243469 13822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:33.243723 13822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:33.248019 13822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40023
I20260620 14:21:33.248098 13889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40023 every 8 connection(s)
I20260620 14:21:33.248430 13822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:21:33.249323 13890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:33.250893 13890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:21:33.251730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13822
I20260620 14:21:33.251858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965139717380-392-0/minicluster-data/master-2/wal/instance
I20260620 14:21:33.252053 392 external_mini_cluster.cc:1428] Setting key 281ae50f90988a36b6af4c1dc93a7d71
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:21:33.262239 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.251646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.0.1:57398)
0620 14:21:33.251825 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.251868 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.252464 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.252767 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.252770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.253166 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.253405 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.253411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.254403 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.254404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.255457 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.255460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.256096 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.256097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.256203 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.256758 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.256833 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.258425 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.260464 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.260469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.260471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.260823 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.261037 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.261038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.261038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.261169 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.261498 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.261502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.261666 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.261838 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.261997 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260620 14:21:33.262490 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.252502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.0.98.60:40023)
0620 14:21:33.252687 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.252692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.252710 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:21:33.252760 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.252762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.252781 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.252868 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.253595 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.254669 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.255715 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.256179 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.256232 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.259134 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.259142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.259142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.259149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.261044 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.261437 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.261439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.261440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.261529 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.261725 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.261726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.261726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.261883 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.262042 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.262137 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.262204 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260620 14:21:33.262614 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.251670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.0.98.62:44869)
0620 14:21:33.251786 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.251789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.251961 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:21:33.252591 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.252594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.252606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.252694 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.253506 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.254309 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.255632 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.255977 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.256059 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.258517 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.258519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.258519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.258527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.260367 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.260904 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.260905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.260905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.260966 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.261259 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.261260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.261261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.261412 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.261484 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.262237 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.262487 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:21:33.263065 13890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:33.270074 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.263493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.0.98.61:38091)
0620 14:21:33.263541 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.263543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.263577 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:21:33.263602 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.263603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.263612 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.263665 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.263980 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.264430 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.265107 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.265322 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.265400 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.267298 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.267299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.267300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.267306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.268907 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.269277 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.269278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.269278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.269333 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.269540 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.269540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.269541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.269666 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.269757 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.269903 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.269977 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:21:33.270089 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.263445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38091 (local address 127.0.0.1:44282)
0620 14:21:33.263485 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.263495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.263530 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.263687 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.263688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.263842 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.263904 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.263906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.264519 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.264520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.265014 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.265015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.265408 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.265409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.265485 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.265889 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.265894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.267196 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.269009 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.269010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.269010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.269175 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.269411 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.269412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.269412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.269472 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.269753 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.269754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.269791 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.269845 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.269962 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:21:33.270555 13890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:21:33.270967 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.263338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.0.98.62:44869)
0620 14:21:33.263377 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.263379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.263415 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:21:33.263541 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.263542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.263573 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.263621 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.264253 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.264701 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.265388 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.265635 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.265700 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.268062 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.268063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.268063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.268069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.269545 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.269909 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.269910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.269910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.269990 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.270217 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.270218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.270219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.270372 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.270538 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.270744 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.270852 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:21:33.277570 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.270823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35804)
0620 14:21:33.270888 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.270898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.270938 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.271084 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.271085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.271219 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.271257 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.271258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.272192 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.272193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.272641 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.272642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.273090 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.273091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.273139 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.273539 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.273544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.274774 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.276505 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.276506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.276506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.276702 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.276947 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.276948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.276948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.277038 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.277330 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.277331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.277366 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.277421 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.277508 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:21:33.277928 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.270857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.0.98.60:40023)
0620 14:21:33.270891 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.270893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.270925 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:21:33.270990 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.270991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.271000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.271049 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.271315 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.272092 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.272769 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.272993 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.273090 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.274858 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.274860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.274861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.274868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.276420 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.276803 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.276804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.276804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.276863 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.277116 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.277117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.277117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.277249 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.277339 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.277456 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.277562 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":17}
I20260620 14:21:33.279124 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.271614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.0.98.61:38091)
0620 14:21:33.271646 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.271647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.271655 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:21:33.271675 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.271676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.271684 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.271731 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.272241 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.272684 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.273370 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.273673 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.273746 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.276144 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.276145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.276146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.276153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.277986 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.278371 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.278372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.278372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.278434 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.278602 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.278602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.278603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.278733 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.278914 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.278962 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.279032 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:21:33.279646 13890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71: Bootstrap starting.
I20260620 14:21:33.280707 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:21:33.280741 13890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:33.281033 13890 log.cc:824] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:33.281872 13890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71: No bootstrap required, opened a new log
I20260620 14:21:33.284132 13890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.284292 13890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:33.284333 13890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a72bc7797349e7926929ef786c3d71, State: Initialized, Role: FOLLOWER
I20260620 14:21:33.284480 13890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.284819 13890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:33.284832 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.284931 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:21:33.286715 13914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:21:33.286772 13914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:21:33.286912 13822 master_runner.cc:186] Error getting master registration for 127.0.98.62:44869: OK, instance_id { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" instance_seqno: 1781965292949598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:21:33.289578 13822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:21:33.291872 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.282371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.0.98.61:38091)
0620 14:21:33.282405 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.282407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.282529 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:21:33.282588 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.282589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.282599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.282646 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.283342 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.284042 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.286092 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.286917 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.286999 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.288600 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.288602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.288602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.288608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.290311 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.290721 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.290723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.290723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.290782 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.291026 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.291028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.291028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.291176 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.291293 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.291684 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.291761 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:21:33.293362 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.282410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.0.98.62:44869)
0620 14:21:33.282455 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.282457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.282656 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:21:33.282746 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.282747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.282757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.282814 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.283318 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.284017 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.287051 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.287634 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.289591 (+ 1957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.289649 (+ 58us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.289650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.289650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.289655 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.292168 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.292572 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.292574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.292574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.292631 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.292820 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.292821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.292822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.292939 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.293050 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.293191 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.293267 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:21:33.293886 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } attempt: 2
I20260620 14:21:33.294157 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.282238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.0.98.60:40023)
0620 14:21:33.282416 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.282418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.283760 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 14:21:33.283862 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.283863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.283871 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.283897 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.284638 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.285212 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.286035 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.286297 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.286371 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.290617 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.290618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.290618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.290626 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.292355 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.292839 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.292840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.292841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.292897 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.293080 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.293080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.293081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.293238 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.293339 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.293996 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.294072 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:21:33.301012 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.294220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.0.98.60:40023)
0620 14:21:33.294352 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.294355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.294362 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:21:33.294380 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.294381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.294390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.294450 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.294735 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.295433 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.296182 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.296493 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.296568 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.298242 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.298243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.298244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.298250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.299801 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.300200 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.300201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.300201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.300259 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.300493 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.300494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.300494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.300602 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.300680 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.300859 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.300927 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 14:21:33.301043 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.294171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35812)
0620 14:21:33.294217 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.294227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.294262 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.294476 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.294477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.294629 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.294669 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.294671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.295511 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.295511 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.296091 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.296093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.296572 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.296572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.296609 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.297040 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.297046 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.298154 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.299900 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.299901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.299901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.300119 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.300335 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.300336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.300336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.300384 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.300685 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.300686 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.300751 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.300832 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.300926 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:21:33.302326 13740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: Bootstrap starting.
I20260620 14:21:33.302975 13740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:33.303272 13740 log.cc:824] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:33.304129 13740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: No bootstrap required, opened a new log
I20260620 14:21:33.305912 13740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.306027 13740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:33.306051 13740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5222b10fe70e4b60b9a6952cc61c38d3, State: Initialized, Role: FOLLOWER
I20260620 14:21:33.306195 13740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.306505 13740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:33.306490 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.306603 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:21:33.309054 13672 master_runner.cc:186] Error getting master registration for 127.0.98.61:38091: OK, instance_id { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" instance_seqno: 1781965293094694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:21:33.311297 13672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:21:33.312171 13928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:21:33.312227 13928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:21:33.329857 13815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } attempt: 3
Jun 20 14:21:33 dist-test-slave-h6n5 krb5kdc[13659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:21:33.338729 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.330166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40023 (local address 127.0.0.1:35820)
0620 14:21:33.330226 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:33.330241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:33.330294 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:33.330487 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:33.330488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:33.330681 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:33.330733 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.330735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.331668 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.331669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.332557 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:33.332560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.333043 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.333044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:33.333103 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.333670 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:33.333675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:33.335232 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:33.337578 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.337579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.337580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.337806 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.338034 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:33.338035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:33.338036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:33.338104 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:33.338400 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:33.338401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:33.338455 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:21:33.338528 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:21:33.338623 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 14:21:33.338744 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:33.330256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.98.60:40023)
0620 14:21:33.330351 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:33.330353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:33.330363 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:21:33.330388 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:33.330389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:33.330400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:33.330461 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:33.330815 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.331595 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.332643 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:33.332953 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:33.333028 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:33.335330 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:33.335332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:33.335332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:33.335340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:33.337482 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.337890 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.337892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.337892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.337953 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:33.338183 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:33.338184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:33.338184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:33.338324 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:33.338393 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:33.338572 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:21:33.338648 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:21:33.340575 13815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: Bootstrap starting.
I20260620 14:21:33.341365 13815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:33.341679 13815 log.cc:824] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:33.342581 13815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: No bootstrap required, opened a new log
I20260620 14:21:33.344630 13815 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.344784 13815 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:33.344820 13815 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65562f92c344c38878e2a129276bb3a, State: Initialized, Role: FOLLOWER
I20260620 14:21:33.345480 13815 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.345846 13815 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:33.345808 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.347187 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:21:33.347785 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:21:33.347883 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:21:33.350250 13747 master_runner.cc:263] No leader master found from master f65562f92c344c38878e2a129276bb3a
I20260620 14:21:33.350304 13747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:21:33.447649 13693 tablet.cc:2369] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:21:33.494450 13843 tablet.cc:2369] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:21:33.573134 13903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:33.573297 13903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.573580 13903 leader_election.cc:288] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5222b10fe70e4b60b9a6952cc61c38d3 (127.0.98.62:44869), f65562f92c344c38878e2a129276bb3a (127.0.98.61:38091)
I20260620 14:21:33.573900 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54a72bc7797349e7926929ef786c3d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" is_pre_election: true
I20260620 14:21:33.573966 13789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54a72bc7797349e7926929ef786c3d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65562f92c344c38878e2a129276bb3a" is_pre_election: true
I20260620 14:21:33.574079 13714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a72bc7797349e7926929ef786c3d71 in term 0.
I20260620 14:21:33.574111 13789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a72bc7797349e7926929ef786c3d71 in term 0.
I20260620 14:21:33.574327 13840 leader_election.cc:302] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a72bc7797349e7926929ef786c3d71, f65562f92c344c38878e2a129276bb3a; no voters:
I20260620 14:21:33.574467 13903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:21:33.574527 13903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:21:33.574555 13903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:33.575356 13903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.575500 13903 leader_election.cc:288] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [CANDIDATE]: Term 1 election: Requested vote from peers 5222b10fe70e4b60b9a6952cc61c38d3 (127.0.98.62:44869), f65562f92c344c38878e2a129276bb3a (127.0.98.61:38091)
I20260620 14:21:33.575693 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54a72bc7797349e7926929ef786c3d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5222b10fe70e4b60b9a6952cc61c38d3"
I20260620 14:21:33.575690 13789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54a72bc7797349e7926929ef786c3d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65562f92c344c38878e2a129276bb3a"
I20260620 14:21:33.575795 13789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:33.575796 13714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:33.576517 13714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a72bc7797349e7926929ef786c3d71 in term 1.
I20260620 14:21:33.576530 13789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a72bc7797349e7926929ef786c3d71 in term 1.
I20260620 14:21:33.576710 13839 leader_election.cc:302] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5222b10fe70e4b60b9a6952cc61c38d3, 54a72bc7797349e7926929ef786c3d71; no voters:
I20260620 14:21:33.576826 13903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:33.576997 13903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [term 1 LEADER]: Becoming Leader. State: Replica: 54a72bc7797349e7926929ef786c3d71, State: Running, Role: LEADER
I20260620 14:21:33.577105 13903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } }
I20260620 14:21:33.577529 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54a72bc7797349e7926929ef786c3d71. Latest consensus state: current_term: 1 leader_uuid: "54a72bc7797349e7926929ef786c3d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.577622 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:33.577811 13946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:33.578003 13946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:33.579367 13789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [term 1 FOLLOWER]: Refusing update from remote peer 54a72bc7797349e7926929ef786c3d71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:33.579663 13714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [term 1 FOLLOWER]: Refusing update from remote peer 54a72bc7797349e7926929ef786c3d71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:33.579689 13903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:33.579906 13903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:33.580991 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54a72bc7797349e7926929ef786c3d71. Latest consensus state: current_term: 1 leader_uuid: "54a72bc7797349e7926929ef786c3d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.581086 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:21:33.581156 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54a72bc7797349e7926929ef786c3d71. Latest consensus state: current_term: 1 leader_uuid: "54a72bc7797349e7926929ef786c3d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.581238 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:21:33.581959 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54a72bc7797349e7926929ef786c3d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.582051 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:33.582132 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54a72bc7797349e7926929ef786c3d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.582228 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:21:33.582345 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54a72bc7797349e7926929ef786c3d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.582430 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:33.582477 13948 mvcc.cc:205] Tried to move back new op lower bound from 7298929842498826240 to 7298929842492424192. Current Snapshot: MvccSnapshot[applied={T|T < 7298929842498826240 or (T in {7298929842498826240})}]
I20260620 14:21:33.582373 13947 mvcc.cc:205] Tried to move back new op lower bound from 7298929842498826240 to 7298929842492424192. Current Snapshot: MvccSnapshot[applied={T|T < 7298929842498826240}]
I20260620 14:21:33.582831 13949 mvcc.cc:205] Tried to move back new op lower bound from 7298929842498826240 to 7298929842492424192. Current Snapshot: MvccSnapshot[applied={T|T < 7298929842498826240}]
I20260620 14:21:33.582860 13946 catalog_manager.cc:1347] Generated new cluster ID: 3add8183ca9c4cbc9475e4dabed9df9b
I20260620 14:21:33.582908 13946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:33.583899 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54a72bc7797349e7926929ef786c3d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222b10fe70e4b60b9a6952cc61c38d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "f65562f92c344c38878e2a129276bb3a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38091 } } peers { permanent_uuid: "54a72bc7797349e7926929ef786c3d71" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40023 } } }
I20260620 14:21:33.583986 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:21:33.594100 13946 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:33.594738 13946 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:33.597754 13946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54a72bc7797349e7926929ef786c3d71: Generated new TSK 0
I20260620 14:21:33.597981 13946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:21:34.312666 13928 catalog_manager.cc:1259] Loaded cluster ID: 3add8183ca9c4cbc9475e4dabed9df9b
I20260620 14:21:34.312731 13928 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: loading cluster ID for follower catalog manager: success
I20260620 14:21:34.313604 13928 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: acquiring CA information for follower catalog manager: success
I20260620 14:21:34.313939 13928 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5222b10fe70e4b60b9a6952cc61c38d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:21:34.348323 13942 catalog_manager.cc:1259] Loaded cluster ID: 3add8183ca9c4cbc9475e4dabed9df9b
I20260620 14:21:34.348400 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: loading cluster ID for follower catalog manager: success
I20260620 14:21:34.349184 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: acquiring CA information for follower catalog manager: success
I20260620 14:21:34.349434 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f65562f92c344c38878e2a129276bb3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:21:34.658931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13672
I20260620 14:21:34.663998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13747
I20260620 14:21:34.668983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13822
2026-06-20T14:21:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3006 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:21:34 dist-test-slave-h6n5 krb5kdc[13962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:21:34 dist-test-slave-h6n5 krb5kdc[13962](info): set up 2 sockets
Jun 20 14:21:34 dist-test-slave-h6n5 krb5kdc[13962](info): commencing operation
krb5kdc: starting...
W20260620 14:21:35.719621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:21:35 dist-test-slave-h6n5 krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:21:35.842795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:35.926884 13975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:35.927084 13975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:35.927107 13975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:35.928318 13975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:35.928364 13975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:35.928378 13975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:35.928390 13975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:35.928403 13975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:35.929821 13975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:35.930004 13975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:35.930173 13975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:35.935184 13981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:35.935158 13980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:35.935175 13983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:35.935371 13975 server_base.cc:1034] running on GCE node
I20260620 14:21:35.935689 13975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:35.935993 13975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:35.937188 13975 hybrid_clock.cc:648] HybridClock initialized: now 1781965295937168 us; error 29 us; skew 500 ppm
Jun 20 14:21:35 dist-test-slave-h6n5 krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:35.939669 13975 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:35.940124 13975 webserver.cc:460] Webserver started at http://127.0.98.62:45933/ using document root <none> and password file <none>
I20260620 14:21:35.940376 13975 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:35.940451 13975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:35.940608 13975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:35.941668 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38"
format_stamp: "Formatted at 2026-06-20 14:21:35 on dist-test-slave-h6n5"
server_key: "6ade844772dc79a26c16d30ef12ef911"
server_key_iv: "165edbb1dbb639ec1577104346773737"
server_key_version: "encryptionkey@0"
I20260620 14:21:35.942039 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38"
format_stamp: "Formatted at 2026-06-20 14:21:35 on dist-test-slave-h6n5"
server_key: "6ade844772dc79a26c16d30ef12ef911"
server_key_iv: "165edbb1dbb639ec1577104346773737"
server_key_version: "encryptionkey@0"
I20260620 14:21:35.943356 13975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:21:35.944164 13990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:35.944335 13975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:21:35.944442 13975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38"
format_stamp: "Formatted at 2026-06-20 14:21:35 on dist-test-slave-h6n5"
server_key: "6ade844772dc79a26c16d30ef12ef911"
server_key_iv: "165edbb1dbb639ec1577104346773737"
server_key_version: "encryptionkey@0"
I20260620 14:21:35.944530 13975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:35.954923 13975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:35.956840 13975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:35.957034 13975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:35.960922 13975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35675
I20260620 14:21:35.960989 14042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35675 every 8 connection(s)
I20260620 14:21:35.961375 13975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:35.962266 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:35.964526 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38: Bootstrap starting.
I20260620 14:21:35.965113 14043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:35.965417 14043 log.cc:824] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:35.966125 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38: No bootstrap required, opened a new log
I20260620 14:21:35.967739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 13975
I20260620 14:21:35.967895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:35.967864 14043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260620 14:21:35.968078 14043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:35.968130 14043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e4a6cd1dfa4173a59ef2e9069c0b38, State: Initialized, Role: FOLLOWER
I20260620 14:21:35.968170 392 external_mini_cluster.cc:1428] Setting key 40f4ae6d58f65388463cf924db04d33b
I20260620 14:21:35.968281 14043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260620 14:21:35.968395 14043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:35.968433 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:35.968488 14043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:35.969205 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260620 14:21:35.969342 14043 leader_election.cc:302] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e4a6cd1dfa4173a59ef2e9069c0b38; no voters:
I20260620 14:21:35.969535 14043 leader_election.cc:288] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:35.969551 14048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:35.969710 14048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [term 1 LEADER]: Becoming Leader. State: Replica: 06e4a6cd1dfa4173a59ef2e9069c0b38, State: Running, Role: LEADER
I20260620 14:21:35.969799 14043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:35.969848 14048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260620 14:21:35.970228 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e4a6cd1dfa4173a59ef2e9069c0b38. Latest consensus state: current_term: 1 leader_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } } }
I20260620 14:21:35.970237 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e4a6cd1dfa4173a59ef2e9069c0b38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } } }
I20260620 14:21:35.970415 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:35.970382 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38 [sys.catalog]: This master's current role is: LEADER
W20260620 14:21:35.971680 14063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:21:35.971777 14063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:21:35.971923 14057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:35.972085 14057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:35.973943 14057 catalog_manager.cc:1347] Generated new cluster ID: 6e047b359dce499899cead26110802da
I20260620 14:21:35.973999 14057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:21:35 dist-test-slave-h6n5 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:35.979707 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:35.968621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.0.98.62:35675)
0620 14:21:35.968805 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:35.968815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:35.968856 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:21:35.969212 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:35.969219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:35.969265 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:35.969429 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:35.970247 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:35.971783 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:35.972697 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:35.973218 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:35.973367 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:35.976107 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:35.976110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:35.976111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:35.976121 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:35.978081 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:35.978485 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:35.978488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:35.978488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:35.978598 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:35.978829 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:35.978830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:35.978830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:35.979106 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:35.979252 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:35.979373 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:21:35.979497 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260620 14:21:35.981031 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:21:35.992529 14057 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:35.993218 14057 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:36.002700 14057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e4a6cd1dfa4173a59ef2e9069c0b38: Generated new TSK 0
I20260620 14:21:36.002902 14057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:21:36.035262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 13975
2026-06-20T14:21:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1364 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:21:36 dist-test-slave-h6n5 krb5kdc[14070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:21:36 dist-test-slave-h6n5 krb5kdc[14070](info): set up 2 sockets
Jun 20 14:21:36 dist-test-slave-h6n5 krb5kdc[14070](info): commencing operation
krb5kdc: starting...
W20260620 14:21:37.079996 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:21:37.198664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:37.280260 14083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:37.280527 14083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:37.280565 14083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:37.281774 14083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:37.281859 14083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:37.281884 14083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:37.281905 14083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:37.281934 14083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:37.283442 14083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:37.283752 14083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:37.284001 14083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:37.288992 14088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:37.288980 14089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:37.289003 14091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:37.289177 14083 server_base.cc:1034] running on GCE node
I20260620 14:21:37.289484 14083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:37.289793 14083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:37.290989 14083 hybrid_clock.cc:648] HybridClock initialized: now 1781965297290978 us; error 36 us; skew 500 ppm
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:37.296640 14083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:37.297111 14083 webserver.cc:460] Webserver started at http://127.0.98.62:38821/ using document root <none> and password file <none>
I20260620 14:21:37.297377 14083 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:37.297456 14083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:37.297606 14083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:37.298712 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "a34eabf48c474c258d78286a98e37fe3"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "62814cdf2722beb7117968818c2f2364"
server_key_iv: "aef2c2b37f3186a8c3a246d74c8e8b2e"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.299166 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "a34eabf48c474c258d78286a98e37fe3"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "62814cdf2722beb7117968818c2f2364"
server_key_iv: "aef2c2b37f3186a8c3a246d74c8e8b2e"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.300685 14083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:21:37.301503 14098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.301698 14083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:37.301792 14083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "a34eabf48c474c258d78286a98e37fe3"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "62814cdf2722beb7117968818c2f2364"
server_key_iv: "aef2c2b37f3186a8c3a246d74c8e8b2e"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.301841 14083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:37.312215 14083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:37.314548 14083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:37.314656 14083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:37.318750 14083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45119
I20260620 14:21:37.318821 14150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45119 every 8 connection(s)
I20260620 14:21:37.319170 14083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:37.320166 14151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:37.322458 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Bootstrap starting.
I20260620 14:21:37.323056 14151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:37.323349 14151 log.cc:824] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:37.324067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14083
I20260620 14:21:37.324092 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: No bootstrap required, opened a new log
I20260620 14:21:37.324162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:37.324405 392 external_mini_cluster.cc:1428] Setting key 48ab66f50d08949d3b5342aba605094e
I20260620 14:21:37.326453 14151 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:37.326637 14151 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:37.326683 14151 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34eabf48c474c258d78286a98e37fe3, State: Initialized, Role: FOLLOWER
I20260620 14:21:37.326809 14151 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:37.326911 14151 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:37.326943 14151 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:37.326980 14151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:37.327689 14151 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:37.327791 14151 leader_election.cc:302] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34eabf48c474c258d78286a98e37fe3; no voters:
I20260620 14:21:37.327948 14151 leader_election.cc:288] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:37.328013 14156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:37.328157 14156 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 LEADER]: Becoming Leader. State: Replica: a34eabf48c474c258d78286a98e37fe3, State: Running, Role: LEADER
I20260620 14:21:37.328161 14151 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:37.328305 14156 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:37.328636 14158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a34eabf48c474c258d78286a98e37fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260620 14:21:37.328722 14158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:37.328902 14163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:37.329355 14163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:37.329937 14157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34eabf48c474c258d78286a98e37fe3. Latest consensus state: current_term: 1 leader_uuid: "a34eabf48c474c258d78286a98e37fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260620 14:21:37.330160 14157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:37.331022 14163 catalog_manager.cc:1347] Generated new cluster ID: 665849f1372142fb9961aa94f269d4e7
I20260620 14:21:37.331070 14163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:37.337356 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.325032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.0.98.62:45119)
0620 14:21:37.325225 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.325235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.325268 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:21:37.326021 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.326030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.326093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.326249 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:37.326920 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.328300 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.329465 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.330069 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.330185 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.333141 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:37.333145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:37.333145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:37.333158 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:37.335138 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.335537 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.335540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.335540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.335676 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.335881 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.335882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.335882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.336187 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:37.336334 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.336840 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.336990 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 14:21:37.350106 14163 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:37.350845 14163 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:37.367944 14163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Generated new TSK 0
I20260620 14:21:37.368177 14163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:21:37.398595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:37.485893 14177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:37.486101 14177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:37.486125 14177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:37.487272 14177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:37.487315 14177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:37.487342 14177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:37.488873 14177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:37.489135 14177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:37.489387 14177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:37.494451 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:37.494462 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:37.494647 14177 server_base.cc:1034] running on GCE node
W20260620 14:21:37.494509 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:37.494953 14177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:37.495179 14177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:37.496323 14177 hybrid_clock.cc:648] HybridClock initialized: now 1781965297496311 us; error 29 us; skew 500 ppm
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:37.499039 14177 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:37.499497 14177 webserver.cc:460] Webserver started at http://127.0.98.1:36117/ using document root <none> and password file <none>
I20260620 14:21:37.499773 14177 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:37.499854 14177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:37.500021 14177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:37.501175 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "7e6fb056b7284b40b160bf371a415ac8"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "630e33f816200de6f94a4a927eb8f9b0"
server_key_iv: "b8f3422f915c52cb591da56d7af2a21f"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.501579 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7e6fb056b7284b40b160bf371a415ac8"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "630e33f816200de6f94a4a927eb8f9b0"
server_key_iv: "b8f3422f915c52cb591da56d7af2a21f"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.502936 14177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:21:37.503834 14192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.504220 14177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:37.504325 14177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "7e6fb056b7284b40b160bf371a415ac8"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "630e33f816200de6f94a4a927eb8f9b0"
server_key_iv: "b8f3422f915c52cb591da56d7af2a21f"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.504444 14177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:37.520941 14177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:37.523535 14177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:37.523751 14177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:37.524008 14177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:37.524345 14177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:37.524407 14177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.524451 14177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:37.524510 14177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.530694 14177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45913
I20260620 14:21:37.530774 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45913 every 8 connection(s)
I20260620 14:21:37.531136 14177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:37.534736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14177
I20260620 14:21:37.534886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:21:37.535099 392 external_mini_cluster.cc:1428] Setting key 492419d23c0a27ccd36060b85492d39a
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:37.542150 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.531943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57209 (local address 127.0.98.62:45119)
0620 14:21:37.532041 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.532043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.532175 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:21:37.532685 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.532686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.532698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.532763 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:37.533762 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.534279 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.535887 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.536222 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.536298 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.538812 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:37.538814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:37.538814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:37.538822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:37.540611 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.541059 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.541061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.541061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.541132 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.541441 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.541442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.541442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.541608 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:37.541688 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.541944 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.542039 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:21:37.542186 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.531878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:57209)
0620 14:21:37.532042 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:37.532080 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:37.532567 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:37.532820 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:37.532823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:37.533443 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:37.533637 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.533642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.534380 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.534381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.535787 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.535790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.536340 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.536341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.536454 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.537093 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:37.537101 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:37.538711 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:37.540708 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:37.540714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:37.540716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:37.540969 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:37.541242 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:37.541243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:37.541243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:37.541356 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:37.541688 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:37.541692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:37.541808 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 14:21:37.541894 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:21:37.542022 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":105,"threads_started":1}
I20260620 14:21:37.542856 14306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260620 14:21:37.542970 14306 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:37.543179 14306 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260620 14:21:37.543814 14115 ts_manager.cc:194] Registered new tserver with Master: 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:21:37.544535 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:21:37.562103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:37.644124 14311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:37.644332 14311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:37.644364 14311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:37.645447 14311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:37.645483 14311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:37.645507 14311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:37.646857 14311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:37.647046 14311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:37.647217 14311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:37.652272 14319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:37.652272 14317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:37.652431 14311 server_base.cc:1034] running on GCE node
W20260620 14:21:37.652356 14316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:37.652661 14311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:37.652870 14311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:37.654016 14311 hybrid_clock.cc:648] HybridClock initialized: now 1781965297653992 us; error 41 us; skew 500 ppm
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:37.656836 14311 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:37.657315 14311 webserver.cc:460] Webserver started at http://127.0.98.2:40521/ using document root <none> and password file <none>
I20260620 14:21:37.657543 14311 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:37.657591 14311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:37.657712 14311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:37.658787 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd5c0310c485428cb2e46c6d04ea9cdf"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "57d740e1c8bae5ab26b9f645cb700e66"
server_key_iv: "c7becfc8069627eedcc54e51ebbe9023"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.659162 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd5c0310c485428cb2e46c6d04ea9cdf"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "57d740e1c8bae5ab26b9f645cb700e66"
server_key_iv: "c7becfc8069627eedcc54e51ebbe9023"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.660663 14311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:21:37.661571 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.661808 14311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:37.661883 14311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "fd5c0310c485428cb2e46c6d04ea9cdf"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "57d740e1c8bae5ab26b9f645cb700e66"
server_key_iv: "c7becfc8069627eedcc54e51ebbe9023"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.661967 14311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:37.685770 14311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:37.688294 14311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:37.688417 14311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:37.688722 14311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:37.689065 14311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:37.689101 14311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.689124 14311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:37.689137 14311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.695417 14311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36883
I20260620 14:21:37.695502 14439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36883 every 8 connection(s)
I20260620 14:21:37.695844 14311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:37.697398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14311
I20260620 14:21:37.697528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:21:37.697731 392 external_mini_cluster.cc:1428] Setting key 7dfd6acbe290cf810c93dc6fe15a244c
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:21:37.707422 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.696625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60665 (local address 127.0.98.62:45119)
0620 14:21:37.696714 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.696716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.696897 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:21:37.697630 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.697632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.697644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.697717 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:37.699012 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.699511 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.700762 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.701107 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.701194 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.703968 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:37.703970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:37.703973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:37.703985 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:37.705580 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.706201 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.706202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.706202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.706264 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.706644 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.706645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.706646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.706821 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:37.706892 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.707204 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.707278 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:21:37.707697 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.696571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:60665)
0620 14:21:37.696750 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:37.696785 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:37.697374 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:37.697802 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:37.697806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:37.698610 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:37.698902 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.698908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.699677 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.699678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.700644 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.700647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.701212 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.701213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.701372 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.702123 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:37.702136 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:37.703838 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:37.705701 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:37.705709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:37.705713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:37.706098 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:37.706367 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:37.706368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:37.706368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:37.706521 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:37.706900 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:37.706906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:37.707042 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:21:37.707136 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:21:37.707265 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":112,"threads_started":1}
I20260620 14:21:37.708341 14440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260620 14:21:37.708451 14440 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:37.708606 14440 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260620 14:21:37.709024 14115 ts_manager.cc:194] Registered new tserver with Master: fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:21:37.709537 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:21:37.726238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:37.810169 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:37.810436 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:37.810478 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:37.811771 14445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:37.811846 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:37.811894 14445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:37.813388 14445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:37.813665 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:37.813903 14445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:37.819090 14450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:37.819078 14453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:37.819077 14451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:37.819258 14445 server_base.cc:1034] running on GCE node
I20260620 14:21:37.819628 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:37.819905 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:37.821206 14445 hybrid_clock.cc:648] HybridClock initialized: now 1781965297821190 us; error 31 us; skew 500 ppm
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:37.823817 14445 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:37.824329 14445 webserver.cc:460] Webserver started at http://127.0.98.3:32891/ using document root <none> and password file <none>
I20260620 14:21:37.824579 14445 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:37.824651 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:37.824807 14445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:37.825881 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6a346541ad9487d96caa9de459d9efc"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "73a163bdbb8a2bb50a1b8e97a748af19"
server_key_iv: "7957ea14221cfe8e911bfc1dac0fa3d8"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.826265 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a346541ad9487d96caa9de459d9efc"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "73a163bdbb8a2bb50a1b8e97a748af19"
server_key_iv: "7957ea14221cfe8e911bfc1dac0fa3d8"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.827685 14445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:21:37.828563 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.828775 14445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:37.828869 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "f6a346541ad9487d96caa9de459d9efc"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "73a163bdbb8a2bb50a1b8e97a748af19"
server_key_iv: "7957ea14221cfe8e911bfc1dac0fa3d8"
server_key_version: "encryptionkey@0"
I20260620 14:21:37.828965 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:37.841665 14445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:37.843600 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:37.843734 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:37.843918 14445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:37.844237 14445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:37.844296 14445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.844337 14445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:37.844357 14445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:37.850524 14445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35807
I20260620 14:21:37.850574 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35807 every 8 connection(s)
I20260620 14:21:37.850914 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:37.851747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14445
I20260620 14:21:37.851850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:21:37.852032 392 external_mini_cluster.cc:1428] Setting key 598b499791a0019f2031a4bd8d628533
Jun 20 14:21:37 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:37.860899 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.851849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37889 (local address 127.0.98.62:45119)
0620 14:21:37.851905 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.851908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.852118 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:21:37.852803 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.852804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.852816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.852862 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:37.853966 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.854496 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.855524 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.855765 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.855866 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.858071 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:37.858072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:37.858073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:37.858080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:37.859342 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.859894 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.859896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.859896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.859952 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.860211 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.860212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.860212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.860325 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:37.860404 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.860675 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.860754 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:21:37.861061 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.851765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:37889)
0620 14:21:37.851972 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:37.852012 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:37.852681 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:37.853004 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:37.853009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:37.853601 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:37.853857 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.853863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.854608 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.854609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.855412 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.855413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.855875 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.855876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.856013 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.856586 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:37.856603 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:37.857972 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:37.859461 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:37.859476 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:37.859479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:37.859770 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:37.860027 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:37.860028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:37.860028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:37.860135 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:37.860402 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:37.860409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:37.860543 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:21:37.860654 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:21:37.860829 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260620 14:21:37.861582 14574 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260620 14:21:37.861707 14574 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:37.861898 14574 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260620 14:21:37.862309 14115 ts_manager.cc:194] Registered new tserver with Master: f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807)
I20260620 14:21:37.862692 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37889
I20260620 14:21:37.863790 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:21:37.871299 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.865188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.0.98.62:45119)
0620 14:21:37.865272 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.865274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.865394 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:21:37.865483 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.865484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.865493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.865548 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:37.866177 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.866644 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.867415 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.867680 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.867772 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.868877 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:37.868878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:37.868879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:37.868885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:37.870044 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.870458 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.870459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.870459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.870513 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:37.870736 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:37.870737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:37.870737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:37.870895 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:37.871012 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.871115 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.871198 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:21:37.872877 14115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:21:37.874468 14115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:21:37.881036 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.876817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45913 (local address 127.0.0.1:50336)
0620 14:21:37.876974 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:37.877035 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:37.877106 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:37.877514 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:37.877517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:37.877520 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:37.877587 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.877589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.878916 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.878917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.879993 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.879995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.880673 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.880758 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.880772 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:21:37.880886 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:21:37.880946 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":51,"threads_started":1}
I20260620 14:21:37.881157 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.877169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.0.98.3:35807)
0620 14:21:37.877347 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.877352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.877362 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:21:37.877431 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.877433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.877450 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.877526 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:37.877742 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.878505 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880021 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.880764 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880883 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.880947 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.881005 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.881049 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260620 14:21:37.881135 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.876817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35807 (local address 127.0.0.1:35982)
0620 14:21:37.877078 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:37.877090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:37.877345 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:37.877607 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:37.877608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:37.877610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:37.877658 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.877659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.878620 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.878621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.879930 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.879933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.880881 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.880938 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.880941 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:21:37.880995 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:21:37.881027 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260620 14:21:37.881425 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.876943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.0.98.1:45913)
0620 14:21:37.877341 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.877345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.877355 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:21:37.877390 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.877391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.877404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.877494 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:37.877671 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.878800 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880063 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.880566 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880662 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.880725 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.881270 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.881336 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":77,"threads_started":1}
I20260620 14:21:37.881614 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.876837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36883 (local address 127.0.0.1:53914)
0620 14:21:37.877447 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:37.877456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:37.877500 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:37.877835 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:37.877836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:37.877838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:37.877878 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.877879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.879090 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.879091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.880032 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:37.880034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.880984 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:37.881057 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.881062 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:21:37.881115 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:21:37.881534 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":96,"threads_started":1}
I20260620 14:21:37.881860 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:37.877111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.0.98.2:36883)
0620 14:21:37.877275 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:37.877280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:37.877615 (+ 335us) server_negotiation.cc:407] Connection header received
0620 14:21:37.877659 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:37.877661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:37.877677 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:37.877745 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:37.877946 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.878991 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880125 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:37.880836 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:37.880897 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:37.880947 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:37.881724 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 14:21:37.881775 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260620 14:21:37.882000 14508 tablet_service.cc:1467] Processing CreateTablet for tablet 1a048dda454542aebde60306797f06ca (DEFAULT_TABLE table=test-table [id=eeafbbde8f9e478ca204cb120923d75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:37.882153 14240 tablet_service.cc:1467] Processing CreateTablet for tablet 1a048dda454542aebde60306797f06ca (DEFAULT_TABLE table=test-table [id=eeafbbde8f9e478ca204cb120923d75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:37.882275 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a048dda454542aebde60306797f06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:37.882401 14240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a048dda454542aebde60306797f06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:37.884438 14374 tablet_service.cc:1467] Processing CreateTablet for tablet 1a048dda454542aebde60306797f06ca (DEFAULT_TABLE table=test-table [id=eeafbbde8f9e478ca204cb120923d75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:37.884793 14374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a048dda454542aebde60306797f06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:37.885259 14591 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Bootstrap starting.
I20260620 14:21:37.885563 14592 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap starting.
I20260620 14:21:37.886060 14591 tablet_bootstrap.cc:654] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:37.886276 14592 tablet_bootstrap.cc:654] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:37.886765 14591 log.cc:824] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:37.886843 14592 log.cc:824] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:37.887676 14592 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: No bootstrap required, opened a new log
I20260620 14:21:37.887748 14592 ts_tablet_manager.cc:1397] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:37.888011 14595 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Bootstrap starting.
I20260620 14:21:37.888082 14591 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: No bootstrap required, opened a new log
I20260620 14:21:37.888139 14591 ts_tablet_manager.cc:1397] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:21:37.888657 14595 tablet_bootstrap.cc:654] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:37.889209 14595 log.cc:824] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:37.889214 14592 raft_consensus.cc:348] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:37.889349 14592 raft_consensus.cc:374] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:37.889371 14592 raft_consensus.cc:729] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6fb056b7284b40b160bf371a415ac8, State: Initialized, Role: FOLLOWER
I20260620 14:21:37.889514 14592 consensus_queue.cc:260] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:37.889799 14592 ts_tablet_manager.cc:1428] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:37.889938 14306 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260620 14:21:37.890033 14595 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: No bootstrap required, opened a new log
I20260620 14:21:37.890130 14595 ts_tablet_manager.cc:1397] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:37.890318 14591 raft_consensus.cc:348] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:37.890604 14591 raft_consensus.cc:374] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:37.890637 14591 raft_consensus.cc:729] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a346541ad9487d96caa9de459d9efc, State: Initialized, Role: FOLLOWER
I20260620 14:21:37.890805 14591 consensus_queue.cc:260] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:37.891085 14591 ts_tablet_manager.cc:1428] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:37.891103 14574 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260620 14:21:37.892338 14595 raft_consensus.cc:348] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:37.892469 14595 raft_consensus.cc:374] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:37.892493 14595 raft_consensus.cc:729] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5c0310c485428cb2e46c6d04ea9cdf, State: Initialized, Role: FOLLOWER
I20260620 14:21:37.892606 14595 consensus_queue.cc:260] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:37.892900 14440 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260620 14:21:37.892901 14595 ts_tablet_manager.cc:1428] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260620 14:21:37.946480 14441 tablet.cc:2369] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:21:38.031905 14307 tablet.cc:2369] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:21:38.101576 14575 tablet.cc:2369] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:21:38.239010 14598 raft_consensus.cc:482] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:38.239212 14598 raft_consensus.cc:504] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:38.239675 14598 leader_election.cc:288] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913), fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883)
I20260620 14:21:38.242507 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.239888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36883 (local address 127.0.98.3:41703)
0620 14:21:38.239945 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.239959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.240002 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.240175 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.240176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.240178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:38.240260 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.240262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.240926 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.240927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.241668 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.241670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.242232 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.242233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.242319 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.242322 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.242388 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.242429 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:21:38.242507 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.239974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41703 (local address 127.0.98.2:36883)
0620 14:21:38.240039 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.240041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.240051 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:21:38.240079 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.240080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.240098 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.240171 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:38.240341 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.240832 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.241775 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.242137 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.242226 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.242237 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.242409 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.242443 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:21:38.242686 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.239958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40979 (local address 127.0.98.1:45913)
0620 14:21:38.240009 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.240011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.240173 (+ 162us) server_negotiation.cc:407] Connection header received
0620 14:21:38.240241 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.240243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.240255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.240325 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:38.240493 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.240959 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.241962 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.242325 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.242413 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.242422 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.242585 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.242626 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:21:38.242686 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.239922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45913 (local address 127.0.98.3:40979)
0620 14:21:38.240078 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.240090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.240182 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.240341 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.240342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.240344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:38.240418 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.240420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.241087 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.241088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.241861 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.241863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.242414 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.242415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.242491 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.242494 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.242568 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.242605 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260620 14:21:38.242951 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a048dda454542aebde60306797f06ca" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" is_pre_election: true
I20260620 14:21:38.243065 14260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a048dda454542aebde60306797f06ca" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6fb056b7284b40b160bf371a415ac8" is_pre_election: true
I20260620 14:21:38.243188 14394 raft_consensus.cc:2454] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 0.
I20260620 14:21:38.243248 14260 raft_consensus.cc:2454] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 0.
I20260620 14:21:38.243458 14463 leader_election.cc:302] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf; no voters:
I20260620 14:21:38.243606 14598 raft_consensus.cc:2781] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:21:38.243659 14598 raft_consensus.cc:482] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:21:38.243681 14598 raft_consensus.cc:3037] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:38.244374 14598 raft_consensus.cc:504] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:38.244534 14598 leader_election.cc:288] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 election: Requested vote from peers 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913), fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883)
I20260620 14:21:38.244671 14260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a048dda454542aebde60306797f06ca" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6fb056b7284b40b160bf371a415ac8"
I20260620 14:21:38.244729 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a048dda454542aebde60306797f06ca" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf"
I20260620 14:21:38.244752 14260 raft_consensus.cc:3037] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:38.244838 14394 raft_consensus.cc:3037] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:38.245536 14260 raft_consensus.cc:2454] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 1.
I20260620 14:21:38.245549 14394 raft_consensus.cc:2454] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 1.
I20260620 14:21:38.245741 14462 leader_election.cc:302] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e6fb056b7284b40b160bf371a415ac8, f6a346541ad9487d96caa9de459d9efc; no voters:
I20260620 14:21:38.245846 14598 raft_consensus.cc:2781] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:38.246001 14598 raft_consensus.cc:686] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 1 LEADER]: Becoming Leader. State: Replica: f6a346541ad9487d96caa9de459d9efc, State: Running, Role: LEADER
I20260620 14:21:38.246126 14598 consensus_queue.cc:237] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:38.246948 14115 catalog_manager.cc:5495] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a346541ad9487d96caa9de459d9efc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f6a346541ad9487d96caa9de459d9efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } health_report { overall_health: UNKNOWN } } }
I20260620 14:21:38.323942 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.321390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.98.3:35807)
0620 14:21:38.321463 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.321465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.321473 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:21:38.321515 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.321516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.321538 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.321637 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:21:38.321804 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.322289 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.323063 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.323282 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.323366 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.323577 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:21:38.323684 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:21:38.323785 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.323809 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.323831 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:21:38.325774 14260 raft_consensus.cc:1261] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Refusing update from remote peer f6a346541ad9487d96caa9de459d9efc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:38.325774 14394 raft_consensus.cc:1261] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Refusing update from remote peer f6a346541ad9487d96caa9de459d9efc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:38.326202 14601 consensus_queue.cc:1035] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:38.326328 14598 consensus_queue.cc:1035] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:38.328660 14604 mvcc.cc:205] Tried to move back new op lower bound from 7298929861939515392 to 7298929861616566272. Current Snapshot: MvccSnapshot[applied={T|T < 7298929861939515392}]
I20260620 14:21:38.328819 14606 mvcc.cc:205] Tried to move back new op lower bound from 7298929861939515392 to 7298929861616566272. Current Snapshot: MvccSnapshot[applied={T|T < 7298929861939515392}]
I20260620 14:21:38.329494 14605 mvcc.cc:205] Tried to move back new op lower bound from 7298929861939515392 to 7298929861616566272. Current Snapshot: MvccSnapshot[applied={T|T < 7298929861939515392 or (T in {7298929861939515392})}]
I20260620 14:21:38.331609 14114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:38.331715 14114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:38.332507 14114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Sending DeleteTablet for 3 replicas of tablet 1a048dda454542aebde60306797f06ca
I20260620 14:21:38.332818 14240 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a048dda454542aebde60306797f06ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50336
I20260620 14:21:38.332846 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a048dda454542aebde60306797f06ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35982
I20260620 14:21:38.332975 14374 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a048dda454542aebde60306797f06ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53914
I20260620 14:21:38.333177 14614 tablet_replica.cc:330] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: stopping tablet replica
I20260620 14:21:38.333185 14616 tablet_replica.cc:330] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: stopping tablet replica
I20260620 14:21:38.333247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14177
I20260620 14:21:38.333273 14616 raft_consensus.cc:2229] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:38.333313 14614 raft_consensus.cc:2229] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:38.333426 14614 raft_consensus.cc:2258] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:38.333927 14616 raft_consensus.cc:2258] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:38.334378 14616 ts_tablet_manager.cc:1905] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:38.336431 14616 ts_tablet_manager.cc:1918] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:38.336508 14616 log.cc:1196] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/1a048dda454542aebde60306797f06ca
I20260620 14:21:38.336750 14616 ts_tablet_manager.cc:1939] T 1a048dda454542aebde60306797f06ca P fd5c0310c485428cb2e46c6d04ea9cdf: Deleting consensus metadata
W20260620 14:21:38.338510 14100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45913 (error 108)
I20260620 14:21:38.339210 14101 catalog_manager.cc:4841] TS fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883): tablet 1a048dda454542aebde60306797f06ca (table test-table [id=eeafbbde8f9e478ca204cb120923d75c]) successfully deleted
I20260620 14:21:38.339363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14311
I20260620 14:21:38.339404 14615 tablet_replica.cc:330] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: stopping tablet replica
I20260620 14:21:38.341188 14615 raft_consensus.cc:2229] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:21:38.341338 14615 raft_consensus.cc:2258] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:38.341814 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.341701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45913 (local address 127.0.0.1:50342)
0620 14:21:38.341747 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.341780 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:21:38.341872 14615 ts_tablet_manager.cc:1905] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:21:38.342010 14100 catalog_manager.cc:4584] TS 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a048dda454542aebde60306797f06ca: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45913: connect: Connection refused (error 111)
I20260620 14:21:38.344007 14615 ts_tablet_manager.cc:1918] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:38.344087 14615 log.cc:1196] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/1a048dda454542aebde60306797f06ca
I20260620 14:21:38.344326 14615 ts_tablet_manager.cc:1939] T 1a048dda454542aebde60306797f06ca P f6a346541ad9487d96caa9de459d9efc: Deleting consensus metadata
I20260620 14:21:38.344915 14102 catalog_manager.cc:4841] TS f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807): tablet 1a048dda454542aebde60306797f06ca (table test-table [id=eeafbbde8f9e478ca204cb120923d75c]) successfully deleted
I20260620 14:21:38.345458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14445
I20260620 14:21:38.351105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14083
I20260620 14:21:38.356251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--webserver_interface=127.0.98.62
--webserver_port=38821
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:38.438053 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:38.438261 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:38.438283 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:38.439471 14618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:38.439513 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:38.439527 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:38.439540 14618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:38.439633 14618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:38.441100 14618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:38.441289 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:38.441457 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:38.450035 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:38.450208 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:38.450213 14618 server_base.cc:1034] running on GCE node
W20260620 14:21:38.450062 14627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:38.450676 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:38.450953 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:38.452098 14618 hybrid_clock.cc:648] HybridClock initialized: now 1781965298452078 us; error 41 us; skew 500 ppm
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:38.458772 14618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:38.459235 14618 webserver.cc:460] Webserver started at http://127.0.98.62:38821/ using document root <none> and password file <none>
I20260620 14:21:38.459474 14618 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:38.459543 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:38.461117 14618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:38.461826 14634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.462031 14618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:38.462141 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "a34eabf48c474c258d78286a98e37fe3"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "62814cdf2722beb7117968818c2f2364"
server_key_iv: "aef2c2b37f3186a8c3a246d74c8e8b2e"
server_key_version: "encryptionkey@0"
I20260620 14:21:38.462563 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:38.473690 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:38.475873 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:38.476034 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:38.480185 14618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45119
I20260620 14:21:38.480217 14686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45119 every 8 connection(s)
I20260620 14:21:38.480618 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:38.480973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14618
I20260620 14:21:38.481277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45913
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=36117
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:38.482503 14687 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:21:38.483489 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Bootstrap starting.
I20260620 14:21:38.491726 14687 log.cc:824] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:38.493767 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:21:38.494114 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Bootstrap complete.
I20260620 14:21:38.496531 14687 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:38.496757 14687 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34eabf48c474c258d78286a98e37fe3, State: Initialized, Role: FOLLOWER
I20260620 14:21:38.496941 14687 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:38.497010 14687 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:38.497069 14687 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:38.497118 14687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:21:38.498136 14687 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:38.498224 14687 leader_election.cc:302] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34eabf48c474c258d78286a98e37fe3; no voters:
I20260620 14:21:38.498456 14687 leader_election.cc:288] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:21:38.498492 14691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:21:38.498610 14691 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 LEADER]: Becoming Leader. State: Replica: a34eabf48c474c258d78286a98e37fe3, State: Running, Role: LEADER
I20260620 14:21:38.498725 14687 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:38.498705 14691 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:38.499022 14691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34eabf48c474c258d78286a98e37fe3. Latest consensus state: current_term: 2 leader_uuid: "a34eabf48c474c258d78286a98e37fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260620 14:21:38.499092 14691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:38.499243 14692 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a34eabf48c474c258d78286a98e37fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260620 14:21:38.499322 14696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:38.499372 14692 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:38.500191 14696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:38.500308 14696 catalog_manager.cc:1259] Loaded cluster ID: 665849f1372142fb9961aa94f269d4e7
I20260620 14:21:38.500345 14696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:38.500579 14696 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:21:38.504243 14696 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:21:38.506644 14696 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:38.506834 14696 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Loaded TSK: 0
I20260620 14:21:38.507053 14696 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:21:38.564534 14689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:38.564747 14689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:38.564786 14689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:38.566079 14689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:38.566148 14689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:38.566190 14689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:38.567741 14689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36117
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:38.567997 14689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:38.568222 14689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:38.573153 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:38.573153 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:38.573297 14689 server_base.cc:1034] running on GCE node
W20260620 14:21:38.573149 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:38.573786 14689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:38.574133 14689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:38.575316 14689 hybrid_clock.cc:648] HybridClock initialized: now 1781965298575296 us; error 37 us; skew 500 ppm
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:38.577886 14689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:38.578394 14689 webserver.cc:460] Webserver started at http://127.0.98.1:36117/ using document root <none> and password file <none>
I20260620 14:21:38.578632 14689 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:38.578686 14689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:38.580122 14689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:21:38.580834 14724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.581059 14689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:38.581166 14689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "7e6fb056b7284b40b160bf371a415ac8"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "630e33f816200de6f94a4a927eb8f9b0"
server_key_iv: "b8f3422f915c52cb591da56d7af2a21f"
server_key_version: "encryptionkey@0"
I20260620 14:21:38.581516 14689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:38.600677 14689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:38.602972 14689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:38.603147 14689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:38.603381 14689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:38.603973 14731 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:21:38.605888 14689 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:21:38.605954 14689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:21:38.605996 14689 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:21:38.606622 14689 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:21:38.606681 14689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:21:38.606745 14731 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap starting.
I20260620 14:21:38.613025 14689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45913
I20260620 14:21:38.613231 14838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45913 every 8 connection(s)
I20260620 14:21:38.613461 14689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:38.616745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14689
I20260620 14:21:38.617038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36883
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=40521
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:38.620245 14731 log.cc:824] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:38.623414 14731 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:21:38.623843 14731 tablet_bootstrap.cc:492] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap complete.
I20260620 14:21:38.625102 14731 ts_tablet_manager.cc:1397] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20260620 14:21:38.625456 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.614212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:55909)
0620 14:21:38.614416 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.614461 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.614936 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.615469 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.615473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.615795 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:38.615970 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.615974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.617235 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.617236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.618052 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.618055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.618932 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.618934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.619098 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.619735 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:38.619742 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:38.621423 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:38.623705 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.623710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.623712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.624022 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.624300 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.624301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.624301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.624424 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.624843 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:38.624848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:38.625007 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.625089 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.625260 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260620 14:21:38.626031 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.614266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55909 (local address 127.0.98.62:45119)
0620 14:21:38.615116 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.615123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.615147 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:21:38.615243 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.615246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.615285 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.615425 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:38.616103 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.617071 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.618173 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.618802 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.619392 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.621553 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:38.621558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:38.621558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:38.621570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:38.623609 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.624117 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.624120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.624120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.624231 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.624492 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.624492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.624493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.624767 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:38.625618 (+ 851us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.625698 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.625826 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":108,"threads_started":1}
I20260620 14:21:38.626652 14839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260620 14:21:38.626746 14839 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:38.626919 14839 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260620 14:21:38.627156 14731 raft_consensus.cc:348] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:38.627516 14731 raft_consensus.cc:729] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6fb056b7284b40b160bf371a415ac8, State: Initialized, Role: FOLLOWER
I20260620 14:21:38.627681 14651 ts_manager.cc:194] Registered new tserver with Master: 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913)
I20260620 14:21:38.627679 14731 consensus_queue.cc:260] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } }
I20260620 14:21:38.627925 14731 ts_tablet_manager.cc:1428] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:21:38.628813 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55909
I20260620 14:21:38.629853 14839 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:21:38.639195 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.628265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45913 (local address 127.0.0.1:50346)
0620 14:21:38.629107 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.629137 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.629198 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.629488 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.629493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.629797 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:38.629880 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.629882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.631252 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.631253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.632286 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.632289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.632854 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.632858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.632943 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.633463 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:38.633469 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:38.635242 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:38.637856 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.637864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.637867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.638146 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.638428 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.638429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.638429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.638500 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.638810 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:38.638822 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:38.638876 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.638962 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.639086 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":91,"threads_started":1}
I20260620 14:21:38.639214 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.628512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.0.98.1:45913)
0620 14:21:38.629331 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.629335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.629344 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:21:38.629376 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.629378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.629391 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.629460 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:38.629990 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.631157 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.632360 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.632771 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.632853 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.635346 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:38.635352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:38.635353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:38.635360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:38.637566 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.638262 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.638266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.638266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.638338 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.638577 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.638578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.638578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.638737 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:38.638816 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.639025 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.639114 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":64,"threads_started":1}
I20260620 14:21:38.640014 14773 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a048dda454542aebde60306797f06ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50346
I20260620 14:21:38.640595 14848 tablet_replica.cc:330] stopping tablet replica
I20260620 14:21:38.640693 14848 raft_consensus.cc:2229] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:38.640756 14848 raft_consensus.cc:2258] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:38.641081 14848 ts_tablet_manager.cc:1905] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:38.643683 14848 ts_tablet_manager.cc:1918] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:38.643754 14848 log.cc:1196] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/1a048dda454542aebde60306797f06ca
I20260620 14:21:38.643973 14848 ts_tablet_manager.cc:1939] T 1a048dda454542aebde60306797f06ca P 7e6fb056b7284b40b160bf371a415ac8: Deleting consensus metadata
I20260620 14:21:38.644472 14636 catalog_manager.cc:4841] TS 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913): tablet 1a048dda454542aebde60306797f06ca (table test-table [id=eeafbbde8f9e478ca204cb120923d75c]) successfully deleted
W20260620 14:21:38.708468 14843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:38.708729 14843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:38.708771 14843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:38.709970 14843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:38.710052 14843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:38.710099 14843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:38.711624 14843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40521
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:38.711908 14843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:38.712149 14843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:38.717303 14856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:38.717383 14853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:38.717304 14854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:38.717465 14843 server_base.cc:1034] running on GCE node
I20260620 14:21:38.717832 14843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:38.718112 14843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:38.719282 14843 hybrid_clock.cc:648] HybridClock initialized: now 1781965298719271 us; error 30 us; skew 500 ppm
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:38.721891 14843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:38.722308 14843 webserver.cc:460] Webserver started at http://127.0.98.2:40521/ using document root <none> and password file <none>
I20260620 14:21:38.722558 14843 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:38.722617 14843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:38.724052 14843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:38.724665 14863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.724898 14843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:38.724997 14843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "fd5c0310c485428cb2e46c6d04ea9cdf"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "57d740e1c8bae5ab26b9f645cb700e66"
server_key_iv: "c7becfc8069627eedcc54e51ebbe9023"
server_key_version: "encryptionkey@0"
I20260620 14:21:38.725343 14843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:38.733879 14843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:38.736244 14843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:38.736413 14843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:38.736655 14843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:38.736961 14843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:38.736997 14843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.737051 14843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:38.737082 14843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.743067 14843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36883
I20260620 14:21:38.743112 14976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36883 every 8 connection(s)
I20260620 14:21:38.743451 14843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:38.744138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14843
I20260620 14:21:38.744447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35807
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=32891
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:38.754895 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.744360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59069 (local address 127.0.98.62:45119)
0620 14:21:38.744414 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.744416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.744659 (+ 243us) server_negotiation.cc:407] Connection header received
0620 14:21:38.745234 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.745236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.745253 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.745324 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:38.746688 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.747178 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.748850 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.749044 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.749100 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.751814 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:38.751815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:38.751815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:38.751822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:38.753345 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.753859 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.753860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.753861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.753912 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.754245 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.754245 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.754246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.754346 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:38.754398 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.754756 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.754827 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:21:38.755210 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.744290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:59069)
0620 14:21:38.744476 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.744561 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.745114 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.745424 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.745427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.746268 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:38.746573 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.746579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.747286 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.747287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.748758 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.748761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.749138 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.749140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.749305 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.749860 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:38.749871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:38.751702 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:38.753432 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.753440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.753442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.753777 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.754007 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.754008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.754009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.754171 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.754422 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:38.754429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:38.754609 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.754739 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.754921 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260620 14:21:38.755827 14977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260620 14:21:38.755932 14977 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:38.756124 14977 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260620 14:21:38.756518 14651 ts_manager.cc:194] Registered new tserver with Master: fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883)
I20260620 14:21:38.756928 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59069
W20260620 14:21:38.829759 14980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:38.829963 14980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:38.829996 14980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:38.831179 14980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:38.831225 14980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:38.831251 14980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:38.832746 14980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32891
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:38.833032 14980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:38.833281 14980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:38.838399 14985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:38.838356 14986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:38.838425 14988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:38.838506 14980 server_base.cc:1034] running on GCE node
I20260620 14:21:38.838753 14980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:38.839062 14980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:38.840243 14980 hybrid_clock.cc:648] HybridClock initialized: now 1781965298840222 us; error 38 us; skew 500 ppm
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:38.843190 14980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:38.843726 14980 webserver.cc:460] Webserver started at http://127.0.98.3:32891/ using document root <none> and password file <none>
I20260620 14:21:38.843966 14980 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:38.844024 14980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:38.845392 14980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:38.846097 14995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.846307 14980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:21:38.846418 14980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "f6a346541ad9487d96caa9de459d9efc"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "73a163bdbb8a2bb50a1b8e97a748af19"
server_key_iv: "7957ea14221cfe8e911bfc1dac0fa3d8"
server_key_version: "encryptionkey@0"
I20260620 14:21:38.846851 14980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:38.857551 14980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:38.859992 14980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:38.860164 14980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:38.860400 14980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:38.860713 14980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:38.860773 14980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.860832 14980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:38.860862 14980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:38.866961 14980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35807
I20260620 14:21:38.867040 15108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35807 every 8 connection(s)
I20260620 14:21:38.867391 14980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:38.869918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 14980
Jun 20 14:21:38 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:38.877959 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.870365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.0.98.62:45119)
0620 14:21:38.870471 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.870473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.870482 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:21:38.870549 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.870550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.870560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.870609 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:38.871269 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.872031 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.872988 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.873199 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.873277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.874703 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:38.874705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:38.874705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:38.874713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:38.876630 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.877083 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.877084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.877085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.877147 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.877381 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.877381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.877382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.877547 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:38.877724 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.877772 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.877841 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260620 14:21:38.878605 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.868270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50643 (local address 127.0.98.62:45119)
0620 14:21:38.868364 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.868367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.868561 (+ 194us) server_negotiation.cc:407] Connection header received
0620 14:21:38.869019 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.869020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.869030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.869085 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:38.869998 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.870792 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.872201 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.872501 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.872575 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.875289 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:38.875290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:38.875291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:38.875297 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:38.876804 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.877338 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.877339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.877339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.877395 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:38.877705 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:38.877706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:38.877706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:38.877852 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:38.877919 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.878433 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.878507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":60}
I20260620 14:21:38.878867 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.868229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:50643)
0620 14:21:38.868426 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.868462 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.868926 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.869138 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.869141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.869702 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:38.869890 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.869894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.870882 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.870883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.872115 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.872118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.872587 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.872588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.872689 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.873274 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:38.873283 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:38.875194 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:38.876900 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.876906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.876908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.877265 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.877495 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:38.877496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:38.877496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:38.877632 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:38.877934 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:38.877941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:38.878292 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.878392 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.878587 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260620 14:21:38.879316 15109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260620 14:21:38.879429 15109 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:38.879637 15109 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260620 14:21:38.880043 14651 ts_manager.cc:194] Registered new tserver with Master: f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807)
I20260620 14:21:38.880362 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50643
I20260620 14:21:38.889794 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:21:38.892843 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.890225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.98.62:45119)
0620 14:21:38.890305 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.890312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.890373 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:21:38.890429 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.890430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.890467 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.890542 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:21:38.890681 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.891192 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.892096 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.892272 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.892327 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.892515 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:21:38.892611 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:21:38.892674 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.892772 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.892787 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:21:38.893196 14651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:21:38.894856 14651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:21:38.897557 14773 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1e56df29245e7b62f0ada8eaec241 (DEFAULT_TABLE table=test-table [id=2000ccc7eb5e4c13b36003dda546e80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:38.897786 14773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1e56df29245e7b62f0ada8eaec241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:38.899385 14731 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap starting.
I20260620 14:21:38.899761 14731 tablet_bootstrap.cc:654] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:38.900413 14731 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: No bootstrap required, opened a new log
I20260620 14:21:38.900488 14731 ts_tablet_manager.cc:1397] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:38.900856 14731 raft_consensus.cc:348] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:38.900950 14731 raft_consensus.cc:374] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:38.901005 14731 raft_consensus.cc:729] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6fb056b7284b40b160bf371a415ac8, State: Initialized, Role: FOLLOWER
I20260620 14:21:38.901046 14731 consensus_queue.cc:260] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:38.901130 14731 ts_tablet_manager.cc:1428] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:38.901444 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.897069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35807 (local address 127.0.0.1:36012)
0620 14:21:38.897125 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.897135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.897174 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.897688 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.897689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.897691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:38.897739 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.897740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.899100 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.899101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.900219 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.900222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.901185 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.901186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.901260 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.901285 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.901319 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.901369 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:21:38.901762 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.897074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36883 (local address 127.0.0.1:53930)
0620 14:21:38.897529 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:38.897539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:38.897600 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:38.897790 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:38.897791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:38.897793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:38.897858 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.897860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.898903 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.898904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.900470 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:38.900473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.901463 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.901464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:38.901539 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.901544 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:21:38.901590 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:21:38.901629 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":325,"thread_start_us":69,"threads_started":1}
I20260620 14:21:38.901772 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.897260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.0.98.3:35807)
0620 14:21:38.897456 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.897462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.897473 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:21:38.897513 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.897515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.897536 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.897682 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:38.897895 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.898960 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.900310 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.901083 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.901554 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.901620 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.901651 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.901692 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260620 14:21:38.902024 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:38.897200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.98.2:36883)
0620 14:21:38.897457 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:38.897462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:38.897613 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:21:38.897697 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:38.897700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:38.897717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:38.897789 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:38.897945 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.898783 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.900560 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:38.901290 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:38.901350 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:38.901407 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:38.901919 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 14:21:38.901964 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":125,"threads_started":1}
I20260620 14:21:38.902575 15043 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1e56df29245e7b62f0ada8eaec241 (DEFAULT_TABLE table=test-table [id=2000ccc7eb5e4c13b36003dda546e80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:38.902848 15043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1e56df29245e7b62f0ada8eaec241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:38.902755 14911 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1e56df29245e7b62f0ada8eaec241 (DEFAULT_TABLE table=test-table [id=2000ccc7eb5e4c13b36003dda546e80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:38.903090 14911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1e56df29245e7b62f0ada8eaec241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:38.905786 15119 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Bootstrap starting.
I20260620 14:21:38.906064 15120 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Bootstrap starting.
I20260620 14:21:38.906489 15119 tablet_bootstrap.cc:654] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:38.906669 15120 tablet_bootstrap.cc:654] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:38.907120 15119 log.cc:824] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:38.907436 15120 log.cc:824] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:38.907917 15119 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: No bootstrap required, opened a new log
I20260620 14:21:38.907972 15119 ts_tablet_manager.cc:1397] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:38.908175 15120 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: No bootstrap required, opened a new log
I20260620 14:21:38.908260 15120 ts_tablet_manager.cc:1397] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:21:38.909411 15119 raft_consensus.cc:348] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:38.909556 15119 raft_consensus.cc:374] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:38.909578 15119 raft_consensus.cc:729] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a346541ad9487d96caa9de459d9efc, State: Initialized, Role: FOLLOWER
I20260620 14:21:38.909600 15120 raft_consensus.cc:348] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:38.909709 15119 consensus_queue.cc:260] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:38.909785 15120 raft_consensus.cc:374] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:38.909832 15120 raft_consensus.cc:729] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5c0310c485428cb2e46c6d04ea9cdf, State: Initialized, Role: FOLLOWER
I20260620 14:21:38.909992 15119 ts_tablet_manager.cc:1428] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:38.909955 15120 consensus_queue.cc:260] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:38.910122 15109 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260620 14:21:38.910247 15120 ts_tablet_manager.cc:1428] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:21:38.910276 14977 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
W20260620 14:21:38.994019 14978 tablet.cc:2369] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:21:39.059442 15123 raft_consensus.cc:482] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:39.059631 15123 raft_consensus.cc:504] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:39.060074 15123 leader_election.cc:288] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883), 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913)
I20260620 14:21:39.063212 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.060337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36883 (local address 127.0.98.3:46707)
0620 14:21:39.060405 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.060419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:39.060465 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:39.060680 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:39.060681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:39.060683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:39.060762 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:39.060764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.061522 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.061523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:39.062316 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:39.062319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.062936 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.062937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:39.063023 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:39.063027 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:39.063087 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:21:39.063128 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:21:39.063241 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.060439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46707 (local address 127.0.98.2:36883)
0620 14:21:39.060514 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:39.060516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:39.060528 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:21:39.060560 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:39.060561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:39.060575 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:39.060650 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:39.060856 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.061420 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.062433 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.062847 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.062908 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:39.062917 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:39.063117 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:21:39.063159 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:21:39.063405 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.060519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38405 (local address 127.0.98.1:45913)
0620 14:21:39.060565 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:39.060567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:39.060681 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:21:39.060710 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:39.060711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:39.060741 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:39.060830 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:39.061057 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.061573 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.062534 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.063022 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.063095 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:39.063169 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:39.063272 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:21:39.063319 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:21:39.063560 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.060337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45913 (local address 127.0.98.3:38405)
0620 14:21:39.060558 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.060569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:39.060640 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:39.060870 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:39.060870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:39.060872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:39.060981 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:39.060983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.061666 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.061667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:39.062434 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:39.062436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.063122 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.063123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:39.063191 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:39.063195 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:39.063253 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:21:39.063439 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":13,"thread_start_us":104,"threads_started":1}
I20260620 14:21:39.063694 14931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1e56df29245e7b62f0ada8eaec241" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" is_pre_election: true
I20260620 14:21:39.063859 14931 raft_consensus.cc:2454] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 0.
I20260620 14:21:39.063932 14792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1e56df29245e7b62f0ada8eaec241" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6fb056b7284b40b160bf371a415ac8" is_pre_election: true
I20260620 14:21:39.064138 14792 raft_consensus.cc:2454] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 0.
I20260620 14:21:39.064129 14998 leader_election.cc:302] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf; no voters:
I20260620 14:21:39.064267 15123 raft_consensus.cc:2781] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:21:39.064327 15123 raft_consensus.cc:482] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:21:39.064352 15123 raft_consensus.cc:3037] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:39.065130 15123 raft_consensus.cc:504] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:39.065295 15123 leader_election.cc:288] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 election: Requested vote from peers fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883), 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913)
I20260620 14:21:39.065440 14792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1e56df29245e7b62f0ada8eaec241" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6fb056b7284b40b160bf371a415ac8"
I20260620 14:21:39.065459 14931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1e56df29245e7b62f0ada8eaec241" candidate_uuid: "f6a346541ad9487d96caa9de459d9efc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf"
I20260620 14:21:39.065531 14931 raft_consensus.cc:3037] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:39.065526 14792 raft_consensus.cc:3037] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:39.066390 14931 raft_consensus.cc:2454] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 1.
I20260620 14:21:39.066390 14792 raft_consensus.cc:2454] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a346541ad9487d96caa9de459d9efc in term 1.
I20260620 14:21:39.066593 14998 leader_election.cc:302] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf; no voters:
I20260620 14:21:39.066717 15123 raft_consensus.cc:2781] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:39.066913 15123 raft_consensus.cc:686] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 1 LEADER]: Becoming Leader. State: Replica: f6a346541ad9487d96caa9de459d9efc, State: Running, Role: LEADER
I20260620 14:21:39.067008 15123 consensus_queue.cc:237] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:39.067837 14651 catalog_manager.cc:5495] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a346541ad9487d96caa9de459d9efc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f6a346541ad9487d96caa9de459d9efc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } health_report { overall_health: UNKNOWN } } }
I20260620 14:21:39.084676 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.082315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.0.98.3:35807)
0620 14:21:39.082373 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:39.082377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:39.082392 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:21:39.082428 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:39.082428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:39.082447 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:39.082512 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:21:39.082625 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.083110 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.084007 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:39.084192 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:39.084247 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:39.084372 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:21:39.084466 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:21:39.084529 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:39.084564 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:21:39.084582 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:21:39.086694 14931 raft_consensus.cc:1261] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Refusing update from remote peer f6a346541ad9487d96caa9de459d9efc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:39.086694 14792 raft_consensus.cc:1261] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Refusing update from remote peer f6a346541ad9487d96caa9de459d9efc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:39.087097 15123 consensus_queue.cc:1035] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:39.087260 15126 consensus_queue.cc:1035] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:39.089736 15128 mvcc.cc:205] Tried to move back new op lower bound from 7298929865055920128 to 7298929864979021824. Current Snapshot: MvccSnapshot[applied={T|T < 7298929865055920128}]
I20260620 14:21:39.090111 15130 mvcc.cc:205] Tried to move back new op lower bound from 7298929865055920128 to 7298929864979021824. Current Snapshot: MvccSnapshot[applied={T|T < 7298929865055920128}]
I20260620 14:21:39.090901 15129 mvcc.cc:205] Tried to move back new op lower bound from 7298929865055920128 to 7298929864979021824. Current Snapshot: MvccSnapshot[applied={T|T < 7298929865055920128 or (T in {7298929865055920128})}]
I20260620 14:21:39.092190 14651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:39.092272 14651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:39.092993 14651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Sending DeleteTablet for 3 replicas of tablet 03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:39.093326 14911 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1e56df29245e7b62f0ada8eaec241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53930
I20260620 14:21:39.093407 14773 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1e56df29245e7b62f0ada8eaec241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50346
I20260620 14:21:39.093408 15043 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1e56df29245e7b62f0ada8eaec241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36012
I20260620 14:21:39.093498 14848 tablet_replica.cc:330] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: stopping tablet replica
I20260620 14:21:39.093565 14848 raft_consensus.cc:2229] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:39.093619 14848 raft_consensus.cc:2258] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:39.093710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14689
I20260620 14:21:39.094220 15138 tablet_replica.cc:330] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: stopping tablet replica
I20260620 14:21:39.094383 15138 raft_consensus.cc:2229] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:39.094552 15138 raft_consensus.cc:2258] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:39.095082 15138 ts_tablet_manager.cc:1905] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:39.096868 15138 ts_tablet_manager.cc:1918] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:39.096997 15138 log.cc:1196] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:39.097370 15138 ts_tablet_manager.cc:1939] T 03e1e56df29245e7b62f0ada8eaec241 P fd5c0310c485428cb2e46c6d04ea9cdf: Deleting consensus metadata
I20260620 14:21:39.097862 14637 catalog_manager.cc:4841] TS fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883): tablet 03e1e56df29245e7b62f0ada8eaec241 (table test-table [id=2000ccc7eb5e4c13b36003dda546e80f]) successfully deleted
I20260620 14:21:39.098043 15139 tablet_replica.cc:330] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: stopping tablet replica
I20260620 14:21:39.098150 15139 raft_consensus.cc:2229] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:21:39.098250 15139 raft_consensus.cc:2258] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:39.098734 15139 ts_tablet_manager.cc:1905] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:21:39.100014 14636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45913 (error 108)
I20260620 14:21:39.100294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14843
I20260620 14:21:39.100901 15139 ts_tablet_manager.cc:1918] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:39.100972 15139 log.cc:1196] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:39.101444 15139 ts_tablet_manager.cc:1939] T 03e1e56df29245e7b62f0ada8eaec241 P f6a346541ad9487d96caa9de459d9efc: Deleting consensus metadata
I20260620 14:21:39.101898 14638 catalog_manager.cc:4841] TS f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807): tablet 03e1e56df29245e7b62f0ada8eaec241 (table test-table [id=2000ccc7eb5e4c13b36003dda546e80f]) successfully deleted
I20260620 14:21:39.104054 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.103881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45913 (local address 127.0.0.1:50362)
0620 14:21:39.103959 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.104011 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:21:39.104236 14636 catalog_manager.cc:4584] TS 7e6fb056b7284b40b160bf371a415ac8 (127.0.98.1:45913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03e1e56df29245e7b62f0ada8eaec241: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45913: connect: Connection refused (error 111)
I20260620 14:21:39.106808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14980
I20260620 14:21:39.111261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 14618
I20260620 14:21:39.116515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--webserver_interface=127.0.98.62
--webserver_port=38821
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:39.196727 15141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:39.196947 15141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:39.196971 15141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:39.198181 15141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:39.198225 15141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:39.198239 15141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:39.198252 15141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:39.198320 15141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:39.199792 15141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:39.200059 15141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:39.200309 15141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:39.205422 15147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:39.205384 15149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:39.205387 15146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:39.205461 15141 server_base.cc:1034] running on GCE node
I20260620 14:21:39.205874 15141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:39.206123 15141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:39.207265 15141 hybrid_clock.cc:648] HybridClock initialized: now 1781965299207254 us; error 28 us; skew 500 ppm
Jun 20 14:21:39 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:39.209715 15141 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:39.210156 15141 webserver.cc:460] Webserver started at http://127.0.98.62:38821/ using document root <none> and password file <none>
I20260620 14:21:39.210332 15141 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:39.210372 15141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:39.211655 15141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:39.212286 15156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.212457 15141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:21:39.212585 15141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "a34eabf48c474c258d78286a98e37fe3"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "62814cdf2722beb7117968818c2f2364"
server_key_iv: "aef2c2b37f3186a8c3a246d74c8e8b2e"
server_key_version: "encryptionkey@0"
I20260620 14:21:39.212936 15141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:39.225328 15141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:39.227087 15141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:39.227259 15141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:39.231498 15141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45119
I20260620 14:21:39.231542 15208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45119 every 8 connection(s)
I20260620 14:21:39.231947 15141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:39.233527 15209 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:21:39.234200 15209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Bootstrap starting.
I20260620 14:21:39.241195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 15141
I20260620 14:21:39.241539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45913
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=36117
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:39.242853 15209 log.cc:824] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:39.245306 15209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:21:39.245625 15209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3: Bootstrap complete.
I20260620 14:21:39.247960 15209 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:39.248183 15209 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34eabf48c474c258d78286a98e37fe3, State: Initialized, Role: FOLLOWER
I20260620 14:21:39.248350 15209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:39.248418 15209 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:39.248481 15209 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:39.248531 15209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:21:39.249536 15209 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:39.249625 15209 leader_election.cc:302] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34eabf48c474c258d78286a98e37fe3; no voters:
I20260620 14:21:39.249835 15209 leader_election.cc:288] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:21:39.249872 15213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:21:39.249982 15213 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [term 3 LEADER]: Becoming Leader. State: Replica: a34eabf48c474c258d78286a98e37fe3, State: Running, Role: LEADER
I20260620 14:21:39.250133 15209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:39.250134 15213 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260620 14:21:39.250561 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a34eabf48c474c258d78286a98e37fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260620 14:21:39.250650 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:39.250972 15219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:21:39.251093 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34eabf48c474c258d78286a98e37fe3. Latest consensus state: current_term: 3 leader_uuid: "a34eabf48c474c258d78286a98e37fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34eabf48c474c258d78286a98e37fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260620 14:21:39.251174 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34eabf48c474c258d78286a98e37fe3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:39.251819 15219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:39.251973 15219 catalog_manager.cc:1259] Loaded cluster ID: 665849f1372142fb9961aa94f269d4e7
I20260620 14:21:39.252168 15219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:21:39.252414 15219 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f600fc0e9ad google::LogMessage::Fail() at ??:0
@ 0x7f600fc1188a google::LogMessage::SendToLog() at ??:0
@ 0x7f600fc0e4de google::LogMessage::Flush() at ??:0
@ 0x7f600fc0fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6017b2f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6017b388b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6011adc0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6011ad1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60129356db start_thread at ??:0
@ 0x7f600e70c61f clone at ??:0
W20260620 14:21:39.335752 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:39.335945 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:39.335968 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:39.337145 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:39.337185 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:39.337212 15211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:39.338634 15211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36117
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:39.338878 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:39.339136 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:39.344188 15248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:39.344192 15246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:39.344513 15211 server_base.cc:1034] running on GCE node
W20260620 14:21:39.344182 15245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:39.344766 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:39.345036 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:39.346208 15211 hybrid_clock.cc:648] HybridClock initialized: now 1781965299346199 us; error 33 us; skew 500 ppm
Jun 20 14:21:39 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:39.348894 15211 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:39.349373 15211 webserver.cc:460] Webserver started at http://127.0.98.1:36117/ using document root <none> and password file <none>
I20260620 14:21:39.349673 15211 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:39.349725 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:39.351177 15211 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:21:39.351965 15255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.352195 15211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:21:39.352299 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "7e6fb056b7284b40b160bf371a415ac8"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "630e33f816200de6f94a4a927eb8f9b0"
server_key_iv: "b8f3422f915c52cb591da56d7af2a21f"
server_key_version: "encryptionkey@0"
I20260620 14:21:39.352671 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:39.379004 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:39.381557 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:39.381727 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:39.382019 15211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:39.382550 15262 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:21:39.384143 15211 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:21:39.384212 15211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:21:39.384249 15211 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:21:39.385170 15211 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:21:39.385217 15211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:21:39.385296 15262 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap starting.
I20260620 14:21:39.392455 15211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45913
I20260620 14:21:39.392632 15369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45913 every 8 connection(s)
I20260620 14:21:39.392891 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:39.396121 15262 log.cc:824] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:39.397663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 15211
I20260620 14:21:39.397923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36883
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=40521
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:21:39.398564 15262 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:21:39.398844 15262 tablet_bootstrap.cc:492] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Bootstrap complete.
I20260620 14:21:39.399943 15262 ts_tablet_manager.cc:1397] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260620 14:21:39.400965 15262 raft_consensus.cc:348] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:39.401218 15262 raft_consensus.cc:729] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6fb056b7284b40b160bf371a415ac8, State: Initialized, Role: FOLLOWER
I20260620 14:21:39.401393 15262 consensus_queue.cc:260] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:39.401677 15262 ts_tablet_manager.cc:1428] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260620 14:21:39.459025 15372 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:21:39.393557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:50799)
0620 14:21:39.393735 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.393778 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:39.394498 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:39.458817 (+ 64319us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
W20260620 14:21:39.459316 15370 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45119 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45119: Client connection negotiation failed: client connection to 127.0.98.62:45119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:21:39.459738 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.459619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:42057)
0620 14:21:39.459667 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.459712 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:21:39.460073 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.459981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:48407)
0620 14:21:39.460029 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.460055 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:21:39.460242 15370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:21:39.482695 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:39.482904 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:39.482929 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:39.484171 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:39.484212 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:39.484262 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:39.485672 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40521
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:39.485919 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:39.486152 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:39.491132 15381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:39.491140 15380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:39.491146 15383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:39.491230 15374 server_base.cc:1034] running on GCE node
I20260620 14:21:39.491588 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:39.491802 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:39.492947 15374 hybrid_clock.cc:648] HybridClock initialized: now 1781965299492936 us; error 27 us; skew 500 ppm
Jun 20 14:21:39 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:39.495347 15374 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:39.495837 15374 webserver.cc:460] Webserver started at http://127.0.98.2:40521/ using document root <none> and password file <none>
I20260620 14:21:39.496078 15374 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:39.496150 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:39.497435 15374 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:39.497987 15390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.498157 15374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:21:39.498256 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "fd5c0310c485428cb2e46c6d04ea9cdf"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "57d740e1c8bae5ab26b9f645cb700e66"
server_key_iv: "c7becfc8069627eedcc54e51ebbe9023"
server_key_version: "encryptionkey@0"
I20260620 14:21:39.498611 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:39.517207 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:39.519347 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:39.519536 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:39.519802 15374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:39.520125 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:39.520183 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.520220 15374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:39.520263 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.526367 15374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36883
I20260620 14:21:39.526456 15503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36883 every 8 connection(s)
I20260620 14:21:39.526836 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:39.528126 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.527686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:43205)
0620 14:21:39.527899 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.527977 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
W20260620 14:21:39.528571 15504 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45119 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45119: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
I20260620 14:21:39.528880 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.528808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:35301)
0620 14:21:39.528837 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.528864 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 14:21:39.529142 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.529092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:60653)
0620 14:21:39.529113 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.529130 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 14:21:39.529299 15504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:21:39.534751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 15374
I20260620 14:21:39.535056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35807
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=32891
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:39.618199 15507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:39.618398 15507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:39.618422 15507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:39.619493 15507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:39.619611 15507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:39.619661 15507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:39.621117 15507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32891
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:39.621296 15507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:39.621465 15507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:39.626573 15512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:39.626591 15515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:39.626699 15507 server_base.cc:1034] running on GCE node
W20260620 14:21:39.626806 15513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:39.627018 15507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:39.627223 15507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:39.628423 15507 hybrid_clock.cc:648] HybridClock initialized: now 1781965299628413 us; error 34 us; skew 500 ppm
Jun 20 14:21:39 dist-test-slave-h6n5 krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:39.631043 15507 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:39.631494 15507 webserver.cc:460] Webserver started at http://127.0.98.3:32891/ using document root <none> and password file <none>
I20260620 14:21:39.631755 15507 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:39.631832 15507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:39.633239 15507 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:39.633857 15522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.634058 15507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:21:39.634158 15507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "f6a346541ad9487d96caa9de459d9efc"
format_stamp: "Formatted at 2026-06-20 14:21:37 on dist-test-slave-h6n5"
server_key: "73a163bdbb8a2bb50a1b8e97a748af19"
server_key_iv: "7957ea14221cfe8e911bfc1dac0fa3d8"
server_key_version: "encryptionkey@0"
I20260620 14:21:39.634533 15507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:39.641819 15507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:39.643699 15507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:39.643853 15507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:39.644074 15507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:39.644371 15507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:39.644426 15507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.644477 15507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:39.644505 15507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:39.650277 15507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35807
I20260620 14:21:39.650332 15635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35807 every 8 connection(s)
I20260620 14:21:39.650696 15507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:21:39.651825 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.651406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:52823)
0620 14:21:39.651607 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.651687 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
W20260620 14:21:39.652242 15636 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45119 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45119: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
I20260620 14:21:39.652511 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.652414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:59321)
0620 14:21:39.652469 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.652495 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:21:39.652755 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:39.652705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:57695)
0620 14:21:39.652725 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:39.652743 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 14:21:39.652863 15636 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:21:39.660316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 15507
I20260620 14:21:40.460863 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:40.460617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:51083)
0620 14:21:40.460744 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:40.460786 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260620 14:21:40.529896 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:40.529668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:53811)
0620 14:21:40.529804 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:40.529844 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":55,"threads_started":1}
I20260620 14:21:40.653441 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:40.653204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:55379)
0620 14:21:40.653346 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:40.653394 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260620 14:21:41.055883 15643 raft_consensus.cc:482] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:41.056013 15643 raft_consensus.cc:504] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:41.056401 15643 leader_election.cc:288] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883), f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807)
Jun 20 14:21:41 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:21:41 dist-test-slave-h6n5 krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:21:41.069448 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:41.056701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37821 (local address 127.0.98.3:35807)
0620 14:21:41.056866 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:41.056873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:41.056905 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:21:41.057457 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:41.057460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:41.057518 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:41.057680 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:41.058809 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.060277 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.061879 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.062509 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.062627 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:41.065539 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:41.065543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:41.065544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:41.065554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:41.067502 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:41.068106 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:41.068109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:41.068109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:41.068202 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:41.068498 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:41.068499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:41.068500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:41.068744 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:41.068860 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:41.069211 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 14:21:41.069346 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20260620 14:21:41.069437 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:41.056593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35807 (local address 127.0.98.1:37821)
0620 14:21:41.056723 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:41.056736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:41.056795 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:41.057763 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:41.057767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:41.058371 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:41.058657 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:41.058664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.060394 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.060395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:41.061777 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:41.061780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.062615 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.062617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:41.062750 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:41.063394 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:41.063415 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:41.065421 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:41.067683 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:41.067690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:41.067692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:41.068008 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:41.068291 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:41.068293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:41.068293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:41.068417 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:41.068883 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:41.068889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:41.069017 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:21:41.069131 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:21:41.069324 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":52,"thread_start_us":71,"threads_started":1}
I20260620 14:21:41.069522 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:41.056716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49699 (local address 127.0.98.2:36883)
0620 14:21:41.056859 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:41.056872 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:41.056898 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:21:41.057457 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:41.057470 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:41.057518 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:41.057680 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:41.058810 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.060277 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.061879 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.062514 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.062627 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:41.065998 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:41.066002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:41.066003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:41.066015 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:41.068069 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:41.068400 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:41.068403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:41.068403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:41.068499 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:41.068724 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:41.068725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:41.068725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:41.068954 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:41.069053 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:41.069255 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:21:41.069406 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":49,"threads_started":1}
I20260620 14:21:41.069552 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:41.056593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36883 (local address 127.0.98.1:49699)
0620 14:21:41.056723 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:41.056736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:41.056795 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:41.057763 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:41.057767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:41.058371 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:41.058657 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:41.058664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.060394 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.060395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:41.061777 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:41.061780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:41.062617 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:41.062617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:41.062750 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:41.063409 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:41.063416 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:41.065814 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:41.068136 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:41.068137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:41.068137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:41.068311 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:41.068578 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:41.068579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:41.068579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:41.068640 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:41.069049 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:41.069050 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:41.069098 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:21:41.069195 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:21:41.069418 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":205,"thread_start_us":59,"threads_started":1}
W20260620 14:21:41.070350 15258 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
W20260620 14:21:41.070890 15259 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:41.070945 15259 leader_election.cc:302] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e6fb056b7284b40b160bf371a415ac8; no voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf
I20260620 14:21:41.071120 15643 raft_consensus.cc:2726] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:21:41.461632 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:41.461484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:36697)
0620 14:21:41.461560 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:41.461591 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:21:41.530738 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:41.530489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:54747)
0620 14:21:41.530636 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:41.530689 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260620 14:21:41.654155 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:41.653930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:33749)
0620 14:21:41.654075 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:41.654112 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260620 14:21:42.462412 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:42.462184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:36949)
0620 14:21:42.462312 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:42.462350 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260620 14:21:42.531472 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:42.531237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:51061)
0620 14:21:42.531391 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:42.531429 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260620 14:21:42.654908 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:42.654701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:54431)
0620 14:21:42.654832 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:42.654871 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260620 14:21:42.794890 15653 raft_consensus.cc:482] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:42.795054 15653 raft_consensus.cc:504] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:42.795274 15653 leader_election.cc:288] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883), f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807)
W20260620 14:21:42.795701 15259 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
W20260620 14:21:42.795773 15258 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:42.795859 15258 leader_election.cc:302] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e6fb056b7284b40b160bf371a415ac8; no voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf
I20260620 14:21:42.795984 15653 raft_consensus.cc:2726] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:21:43.463182 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:43.462919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:46043)
0620 14:21:43.463079 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:43.463135 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260620 14:21:43.532104 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:43.531886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:41221)
0620 14:21:43.532005 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:43.532044 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260620 14:21:43.655797 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:43.655469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:53129)
0620 14:21:43.655702 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:43.655752 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
I20260620 14:21:44.464039 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:44.463821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:59109)
0620 14:21:44.463946 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:44.463988 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":93,"threads_started":1}
I20260620 14:21:44.532753 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:44.532544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:46671)
0620 14:21:44.532677 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:44.532710 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260620 14:21:44.656491 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:44.656243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:37831)
0620 14:21:44.656401 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:44.656446 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260620 14:21:45.025583 15660 raft_consensus.cc:482] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:45.025722 15660 raft_consensus.cc:504] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:45.025930 15660 leader_election.cc:288] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883), f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807)
W20260620 14:21:45.026345 15259 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
W20260620 14:21:45.026418 15258 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:45.026506 15258 leader_election.cc:302] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e6fb056b7284b40b160bf371a415ac8; no voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf
I20260620 14:21:45.026579 15660 raft_consensus.cc:2726] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:21:45.464705 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:45.464483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:39557)
0620 14:21:45.464621 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:45.464659 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260620 14:21:45.533397 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:45.533189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:45117)
0620 14:21:45.533317 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:45.533355 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260620 14:21:45.657264 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:45.657035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:53703)
0620 14:21:45.657175 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:45.657219 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260620 14:21:46.465505 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:46.465311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:37869)
0620 14:21:46.465420 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:46.465459 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":71,"threads_started":1}
I20260620 14:21:46.534090 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:46.533878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:49335)
0620 14:21:46.534008 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:46.534045 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260620 14:21:46.658018 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:46.657798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:43081)
0620 14:21:46.657928 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:46.657973 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260620 14:21:47.466210 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:47.465991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:42831)
0620 14:21:47.466126 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:47.466164 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260620 14:21:47.534799 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:47.534578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:41553)
0620 14:21:47.534718 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:47.534757 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260620 14:21:47.658689 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:47.658457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:51241)
0620 14:21:47.658606 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:47.658646 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260620 14:21:47.837189 15670 raft_consensus.cc:482] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:47.837284 15670 raft_consensus.cc:504] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:47.837499 15670 leader_election.cc:288] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883), f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807)
W20260620 14:21:47.837862 15259 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
W20260620 14:21:47.837925 15258 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:47.837958 15258 leader_election.cc:302] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e6fb056b7284b40b160bf371a415ac8; no voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf
I20260620 14:21:47.838063 15670 raft_consensus.cc:2726] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:21:48.466955 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:48.466722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:37509)
0620 14:21:48.466874 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:48.466910 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260620 14:21:48.535625 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:48.535366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:43829)
0620 14:21:48.535494 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:48.535527 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260620 14:21:48.659477 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:48.659251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:44483)
0620 14:21:48.659386 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:48.659430 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260620 14:21:49.385380 15674 raft_consensus.cc:482] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:49.385488 15674 raft_consensus.cc:504] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd5c0310c485428cb2e46c6d04ea9cdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36883 } } peers { permanent_uuid: "f6a346541ad9487d96caa9de459d9efc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35807 } } peers { permanent_uuid: "7e6fb056b7284b40b160bf371a415ac8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45913 } }
I20260620 14:21:49.385694 15674 leader_election.cc:288] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883), f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807)
W20260620 14:21:49.386119 15258 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5c0310c485428cb2e46c6d04ea9cdf (127.0.98.2:36883): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
W20260620 14:21:49.386237 15259 leader_election.cc:341] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6a346541ad9487d96caa9de459d9efc (127.0.98.3:35807): Not found: Tablet not found: 03e1e56df29245e7b62f0ada8eaec241
I20260620 14:21:49.386281 15259 leader_election.cc:302] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e6fb056b7284b40b160bf371a415ac8; no voters: f6a346541ad9487d96caa9de459d9efc, fd5c0310c485428cb2e46c6d04ea9cdf
I20260620 14:21:49.386353 15674 raft_consensus.cc:2726] T 03e1e56df29245e7b62f0ada8eaec241 P 7e6fb056b7284b40b160bf371a415ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:21:49.467823 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:49.467582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:55291)
0620 14:21:49.467729 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:49.467774 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260620 14:21:49.536314 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:49.536079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:39943)
0620 14:21:49.536231 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:49.536266 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260620 14:21:49.660457 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:49.660214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:49857)
0620 14:21:49.660362 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:49.660408 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260620 14:21:50.468590 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:50.468373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:38469)
0620 14:21:50.468503 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:50.468542 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260620 14:21:50.537019 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:50.536813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:54215)
0620 14:21:50.536942 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:50.536977 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260620 14:21:50.661244 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:50.660983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:58803)
0620 14:21:50.661140 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:50.661199 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260620 14:21:51.469308 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:51.469053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:49645)
0620 14:21:51.469202 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:51.469259 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260620 14:21:51.537765 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:51.537522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:53291)
0620 14:21:51.537673 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:51.537715 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260620 14:21:51.661931 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:51.661711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:45223)
0620 14:21:51.661847 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:51.661886 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260620 14:21:52.470149 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:52.469810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:40707)
0620 14:21:52.470034 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:52.470075 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":178,"threads_started":1}
I20260620 14:21:52.538439 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:52.538198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:56475)
0620 14:21:52.538343 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:52.538383 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260620 14:21:52.662602 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:52.662380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:54071)
0620 14:21:52.662509 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:52.662548 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260620 14:21:53.470883 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:53.470654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:58845)
0620 14:21:53.470799 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:53.470836 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260620 14:21:53.539149 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:53.538952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:55385)
0620 14:21:53.539065 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:53.539102 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260620 14:21:53.663251 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:53.663027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:48575)
0620 14:21:53.663170 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:53.663208 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260620 14:21:54.471604 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:54.471359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:36241)
0620 14:21:54.471472 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:54.471501 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260620 14:21:54.539922 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:54.539702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:42783)
0620 14:21:54.539837 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:54.539879 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260620 14:21:54.660697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 15211
I20260620 14:21:54.664115 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:54.663876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:57291)
0620 14:21:54.664006 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:54.664055 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260620 14:21:54.665748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 15374
I20260620 14:21:54.669865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 15507
2026-06-20T14:21:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18636 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:21:54 dist-test-slave-h6n5 krb5kdc[15695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:21:54 dist-test-slave-h6n5 krb5kdc[15695](info): set up 2 sockets
Jun 20 14:21:54 dist-test-slave-h6n5 krb5kdc[15695](info): commencing operation
krb5kdc: starting...
W20260620 14:21:55.704490 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:21:55 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:21:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:21:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:21:55.822526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:55.902966 15708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:55.903160 15708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:55.903183 15708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:55.904356 15708 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:21:55.904398 15708 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:21:55.904412 15708 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:21:55.904424 15708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:21:55.904436 15708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:55.904449 15708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:55.904508 15708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:21:55.904526 15708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:21:55.906450 15708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:55.906713 15708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:55.906991 15708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:55.912325 15714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:55.912309 15713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:55.912559 15708 server_base.cc:1034] running on GCE node
W20260620 14:21:55.912706 15716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:55.912987 15708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:55.913368 15708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:55.914551 15708 hybrid_clock.cc:648] HybridClock initialized: now 1781965315914513 us; error 56 us; skew 500 ppm
Jun 20 14:21:55 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:55.918629 15708 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:21:55.919210 15708 webserver.cc:460] Webserver started at http://127.0.98.62:40349/ using document root <none> and password file <none>
I20260620 14:21:55.919517 15708 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:55.919620 15708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:55.919785 15708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:55.921206 15708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "65f7daed94244fc8ba5ebfee80763c14"
format_stamp: "Formatted at 2026-06-20 14:21:55 on dist-test-slave-h6n5"
server_key: "f7f25515efca2410f23cf6903de1f9ed"
server_key_iv: "7f7f9c8526b272ed01181c2c5cc72bde"
server_key_version: "encryptionkey@0"
I20260620 14:21:55.921684 15708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "65f7daed94244fc8ba5ebfee80763c14"
format_stamp: "Formatted at 2026-06-20 14:21:55 on dist-test-slave-h6n5"
server_key: "f7f25515efca2410f23cf6903de1f9ed"
server_key_iv: "7f7f9c8526b272ed01181c2c5cc72bde"
server_key_version: "encryptionkey@0"
I20260620 14:21:55.923130 15708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:55.923863 15723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:55.924037 15708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:55.924154 15708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "65f7daed94244fc8ba5ebfee80763c14"
format_stamp: "Formatted at 2026-06-20 14:21:55 on dist-test-slave-h6n5"
server_key: "f7f25515efca2410f23cf6903de1f9ed"
server_key_iv: "7f7f9c8526b272ed01181c2c5cc72bde"
server_key_version: "encryptionkey@0"
I20260620 14:21:55.924238 15708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:55.946810 15708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:55.949242 15708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:55.949389 15708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:55.953261 15708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40047
I20260620 14:21:55.953323 15775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40047 every 8 connection(s)
I20260620 14:21:55.953655 15708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:21:55.954559 15776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:55.956888 15776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Bootstrap starting.
I20260620 14:21:55.957013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 15708
I20260620 14:21:55.957125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:21:55.957340 392 external_mini_cluster.cc:1428] Setting key ddd87f3fc5e00e3ad816dcba17cbd3c7
I20260620 14:21:55.957571 15776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:55.957926 15776 log.cc:824] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:55.958653 15776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: No bootstrap required, opened a new log
I20260620 14:21:55.961019 15776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:21:55.961206 15776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:55.961241 15776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7daed94244fc8ba5ebfee80763c14, State: Initialized, Role: FOLLOWER
I20260620 14:21:55.961354 15776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:21:55.961421 15776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:21:55.961448 15776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:21:55.961489 15776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:55.962095 15776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:21:55.962178 15776 leader_election.cc:302] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65f7daed94244fc8ba5ebfee80763c14; no voters:
I20260620 14:21:55.962369 15776 leader_election.cc:288] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:21:55.962420 15781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:55.962632 15781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 LEADER]: Becoming Leader. State: Replica: 65f7daed94244fc8ba5ebfee80763c14, State: Running, Role: LEADER
I20260620 14:21:55.962693 15776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:21:55.962767 15781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:21:55.963059 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65f7daed94244fc8ba5ebfee80763c14. Latest consensus state: current_term: 1 leader_uuid: "65f7daed94244fc8ba5ebfee80763c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
I20260620 14:21:55.963079 15783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65f7daed94244fc8ba5ebfee80763c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
I20260620 14:21:55.963181 15783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:55.963403 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: This master's current role is: LEADER
I20260620 14:21:55.964345 15785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 14:21:55 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:55.965062 15785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:21:55.966764 15785 catalog_manager.cc:1347] Generated new cluster ID: de793ee03bb942a79c213df2a3ea1789
I20260620 14:21:55.966853 15785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:21:55.969146 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:55.957857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.0.98.62:40047)
0620 14:21:55.958055 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:55.958064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:55.958136 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:21:55.958709 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:55.958718 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:55.958786 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:55.958900 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:55.959785 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:55.960670 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:55.961753 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:55.962280 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:55.962401 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:55.965491 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:55.965495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:55.965495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:55.965506 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:55.967579 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:55.967971 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:55.967973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:55.967974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:55.968069 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:55.968280 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:55.968280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:55.968281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:55.968547 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:55.968680 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:55.968801 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:21:55.968924 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260620 14:21:55.972615 15785 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:21:55.973088 15785 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:21:55.978440 15785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Generated new TSK 0
I20260620 14:21:55.978600 15785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:21:56.006213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:56.088696 15802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:56.088948 15802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:56.088990 15802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:56.090235 15802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:56.090296 15802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:56.090341 15802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:21:56.091813 15802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:56.092082 15802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:56.092350 15802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:56.097378 15808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:56.097378 15810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:56.097432 15807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:56.097631 15802 server_base.cc:1034] running on GCE node
I20260620 14:21:56.097798 15802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:56.097985 15802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:56.099159 15802 hybrid_clock.cc:648] HybridClock initialized: now 1781965316099143 us; error 29 us; skew 500 ppm
Jun 20 14:21:56 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:56.101660 15802 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:21:56.102283 15802 webserver.cc:460] Webserver started at http://127.0.98.1:33145/ using document root <none> and password file <none>
I20260620 14:21:56.102536 15802 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:56.102607 15802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:56.102766 15802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:56.103885 15802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "54ef2cd13f664f1f8dbc452f42340d60"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "b235985c3cdbcc57702e6eb68914538d"
server_key_iv: "722a8d0b9acfe0b82993cfc1da046b0e"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.104271 15802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "54ef2cd13f664f1f8dbc452f42340d60"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "b235985c3cdbcc57702e6eb68914538d"
server_key_iv: "722a8d0b9acfe0b82993cfc1da046b0e"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.105576 15802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:21:56.106323 15817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.106495 15802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:21:56.106599 15802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "54ef2cd13f664f1f8dbc452f42340d60"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "b235985c3cdbcc57702e6eb68914538d"
server_key_iv: "722a8d0b9acfe0b82993cfc1da046b0e"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.106671 15802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:56.124282 15802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:56.127068 15802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:56.127233 15802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:56.127537 15802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:56.127988 15802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:56.128047 15802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.128088 15802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:56.128119 15802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.135025 15802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45865
I20260620 14:21:56.135093 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45865 every 8 connection(s)
I20260620 14:21:56.135424 15802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:21:56.140872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 15802
I20260620 14:21:56.140961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:21:56.141199 392 external_mini_cluster.cc:1428] Setting key 981fb27616f1e67d5a04449ca33e79a7
Jun 20 14:21:56 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:56.145499 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.136201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43223 (local address 127.0.98.62:40047)
0620 14:21:56.136258 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.136260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.136453 (+ 193us) server_negotiation.cc:407] Connection header received
0620 14:21:56.136959 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.136960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.136970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.137021 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:56.137966 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.138483 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.139504 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.139809 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.139872 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.142208 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:56.142210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:56.142211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:56.142219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:56.144020 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.144500 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.144501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.144501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.144570 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.144835 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.144836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.144837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.144959 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:56.145038 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.145306 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.145403 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:21:56.145689 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.136185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:43223)
0620 14:21:56.136347 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.136385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.136871 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.137075 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.137078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.137664 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:56.137874 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.137879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.138574 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.138575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.139400 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.139403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.139889 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.139890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.139991 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.140471 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:56.140480 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:56.142112 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:56.144107 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:56.144113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:56.144114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:56.144412 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:56.144653 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:56.144654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:56.144654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:56.144764 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:56.145027 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:56.145034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:56.145169 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.145285 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.145438 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260620 14:21:56.146214 15931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260620 14:21:56.146344 15931 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:56.146515 15931 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260620 14:21:56.147049 15740 ts_manager.cc:194] Registered new tserver with Master: 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1:45865)
I20260620 14:21:56.147749 15740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43223
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:21:56.166465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:56.248477 15936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:56.248694 15936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:56.248718 15936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:56.249883 15936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:56.249927 15936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:56.249954 15936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:21:56.251472 15936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:56.251754 15936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:56.251995 15936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:56.256985 15944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:56.257128 15941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:56.256994 15942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:56.257781 15936 server_base.cc:1034] running on GCE node
I20260620 14:21:56.258002 15936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:56.258277 15936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:56.259446 15936 hybrid_clock.cc:648] HybridClock initialized: now 1781965316259432 us; error 47 us; skew 500 ppm
Jun 20 14:21:56 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:56.262115 15936 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:21:56.262624 15936 webserver.cc:460] Webserver started at http://127.0.98.2:38907/ using document root <none> and password file <none>
I20260620 14:21:56.262852 15936 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:56.262921 15936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:56.263063 15936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:56.264282 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "204fe3cbac9e4497babca71d31acd60f"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "095a950d882dc7e40f418d5fc3232b78"
server_key_iv: "2b28306b3ec64320ab18e1922d707d59"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.264642 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "204fe3cbac9e4497babca71d31acd60f"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "095a950d882dc7e40f418d5fc3232b78"
server_key_iv: "2b28306b3ec64320ab18e1922d707d59"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.265945 15936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:21:56.266681 15951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.266844 15936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:21:56.266940 15936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "204fe3cbac9e4497babca71d31acd60f"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "095a950d882dc7e40f418d5fc3232b78"
server_key_iv: "2b28306b3ec64320ab18e1922d707d59"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.267024 15936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:56.277256 15936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:56.279644 15936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:56.279803 15936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:56.280031 15936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:56.280335 15936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:56.280388 15936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.280431 15936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:56.280467 15936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.286321 15936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40733
I20260620 14:21:56.286367 16064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40733 every 8 connection(s)
I20260620 14:21:56.286661 15936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:21:56.291373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 15936
I20260620 14:21:56.291477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:21:56.291745 392 external_mini_cluster.cc:1428] Setting key 2370bf27a207edce256ba775e9090152
Jun 20 14:21:56 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:56.296327 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.287429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38121 (local address 127.0.98.62:40047)
0620 14:21:56.287497 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.287499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.287731 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:21:56.288190 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.288191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.288230 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.288288 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:56.289241 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.289738 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.290806 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.290983 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.291033 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.293279 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:56.293280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:56.293281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:56.293288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:56.294823 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.295320 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.295321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.295321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.295369 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.295678 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.295678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.295679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.295780 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:56.295832 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.296187 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.296260 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:21:56.296680 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.287379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:38121)
0620 14:21:56.287594 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.287636 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.288110 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.288377 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.288381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.288954 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:56.289144 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.289149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.289837 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.289838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.290728 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.290732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.291053 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.291054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.291153 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.291721 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:56.291729 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:56.293192 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:56.294906 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:56.294913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:56.294915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:56.295242 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:56.295435 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:56.295436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:56.295436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:56.295612 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:56.295850 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:56.295856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:56.296034 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.296151 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.296340 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260620 14:21:56.297247 16065 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260620 14:21:56.297372 16065 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:56.297560 16065 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260620 14:21:56.297947 15740 ts_manager.cc:194] Registered new tserver with Master: 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733)
I20260620 14:21:56.298280 15740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38121
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:21:56.319469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:21:56.403026 16070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:21:56.403280 16070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:21:56.403321 16070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:21:56.404484 16070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:21:56.404556 16070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:21:56.404600 16070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:21:56.406026 16070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:21:56.406306 16070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:21:56.406534 16070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:21:56.411541 16075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:56.411540 16076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:21:56.411586 16078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:21:56.411800 16070 server_base.cc:1034] running on GCE node
I20260620 14:21:56.412017 16070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:21:56.412266 16070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:21:56.413404 16070 hybrid_clock.cc:648] HybridClock initialized: now 1781965316413381 us; error 40 us; skew 500 ppm
Jun 20 14:21:56 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:21:56.415925 16070 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:21:56.416287 16070 webserver.cc:460] Webserver started at http://127.0.98.3:42185/ using document root <none> and password file <none>
I20260620 14:21:56.416472 16070 fs_manager.cc:359] Metadata directory not provided
I20260620 14:21:56.416512 16070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:21:56.416599 16070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:21:56.417553 16070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "70f07dca7db84b95a05845b800323abc"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "c61c1a8baaf27601c32440316cf1d931"
server_key_iv: "b5314b06d05881512c12925dfcd44dae"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.417846 16070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "70f07dca7db84b95a05845b800323abc"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "c61c1a8baaf27601c32440316cf1d931"
server_key_iv: "b5314b06d05881512c12925dfcd44dae"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.419008 16070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:21:56.419713 16085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.419862 16070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:21:56.419948 16070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "70f07dca7db84b95a05845b800323abc"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "c61c1a8baaf27601c32440316cf1d931"
server_key_iv: "b5314b06d05881512c12925dfcd44dae"
server_key_version: "encryptionkey@0"
I20260620 14:21:56.420001 16070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:21:56.437270 16070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:21:56.439639 16070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:21:56.439770 16070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:21:56.439960 16070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:21:56.440290 16070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:21:56.440326 16070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.440372 16070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:21:56.440398 16070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:21:56.446465 16070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37091
I20260620 14:21:56.446511 16198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37091 every 8 connection(s)
I20260620 14:21:56.446964 16070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:21:56 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:21:56.454562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 16070
I20260620 14:21:56.454696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:21:56.454896 392 external_mini_cluster.cc:1428] Setting key ec3630a180d85c2be90e6a1b46dbf31b
I20260620 14:21:56.456641 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.447824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60463 (local address 127.0.98.62:40047)
0620 14:21:56.447927 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.447929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.448109 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:21:56.448646 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.448647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.448659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.448708 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:56.449684 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.450207 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.451198 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.451406 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.451493 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.453602 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:56.453603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:56.453603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:56.453610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:56.455206 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.455671 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.455672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.455672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.455721 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.456006 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.456007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.456008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.456129 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:56.456205 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.456470 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.456548 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:21:56.456841 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.447741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:60463)
0620 14:21:56.447966 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.448010 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.448525 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.448785 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.448788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.449353 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:21:56.449551 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.449555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.450303 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.450304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.451101 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.451103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.451505 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.451506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.451657 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.452136 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:21:56.452144 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:21:56.453515 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:21:56.455288 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:56.455296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:56.455298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:56.455613 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:56.455803 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:21:56.455804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:21:56.455804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:21:56.455936 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:21:56.456195 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:21:56.456202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:21:56.456342 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.456450 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.456608 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260620 14:21:56.457329 16199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260620 14:21:56.457423 16199 heartbeater.cc:464] Registering TS with master...
I20260620 14:21:56.457608 16199 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260620 14:21:56.457966 15740 ts_manager.cc:194] Registered new tserver with Master: 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
I20260620 14:21:56.458343 15740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60463
I20260620 14:21:56.466797 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:21:56.474630 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.468400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.0.98.62:40047)
0620 14:21:56.468496 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.468498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.468593 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:21:56.468678 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.468679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.468689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.468738 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:21:56.469321 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.469852 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.470601 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.470916 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.471001 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.472262 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:21:56.472264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:21:56.472264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:21:56.472270 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:21:56.473449 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.473833 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.473834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.473835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.473889 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:21:56.474090 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:21:56.474091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:21:56.474092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:21:56.474243 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:21:56.474345 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.474449 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.474530 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:21:56.476281 15740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:21:56.477900 15740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:21:56.484797 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.480387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.0.98.3:37091)
0620 14:21:56.480523 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.480527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.480944 (+ 417us) server_negotiation.cc:407] Connection header received
0620 14:21:56.480984 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.480986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.481001 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.481110 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:56.481349 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.482400 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.483713 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.484413 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.484483 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.484541 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.484672 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.484715 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260620 14:21:56.484789 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.480191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37091 (local address 127.0.0.1:45486)
0620 14:21:56.480858 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.480868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.480917 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.481231 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.481231 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.481233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:56.481276 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.481277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.482502 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.482503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.483608 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.483610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.484500 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.484501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.484554 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.484564 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.484646 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.484697 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":52,"thread_start_us":46,"threads_started":1}
I20260620 14:21:56.485175 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.480303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.0.98.1:45865)
0620 14:21:56.480495 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.480501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.480738 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:21:56.480841 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.480844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.480861 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.480947 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:56.481223 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.482374 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.483900 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.484552 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.484640 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.484700 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.484845 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.485077 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260620 14:21:56.485351 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.480168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45865 (local address 127.0.0.1:35916)
0620 14:21:56.480626 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.480665 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.480755 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.480970 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.481041 (+ 71us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.481044 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:56.481127 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.481129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.482489 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.482490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.483817 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.483820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.484626 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.484627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.484696 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.484699 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.484765 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.484799 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":145,"thread_start_us":48,"threads_started":1}
I20260620 14:21:56.485572 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 0cad0819cdea4d52a73473a6a031fabd (DEFAULT_TABLE table=test-table [id=c710287148a44da4a17372afe6644727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:56.485922 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cad0819cdea4d52a73473a6a031fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:56.485965 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.480191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40733 (local address 127.0.0.1:55708)
0620 14:21:56.480985 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.480994 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.481044 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.481492 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.481493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.481495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:56.481529 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.481530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.482456 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.482457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.483817 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.483820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.485713 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.485714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.485783 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.485787 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.485850 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.485893 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":127,"thread_start_us":34,"threads_started":1}
I20260620 14:21:56.485983 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.480437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.0.98.2:40733)
0620 14:21:56.480586 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.480590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.481171 (+ 581us) server_negotiation.cc:407] Connection header received
0620 14:21:56.481223 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.481225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.481240 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.481313 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:56.481633 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.482350 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.484736 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.485629 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.485714 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.485769 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.485863 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.485899 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 14:21:56.485996 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 0cad0819cdea4d52a73473a6a031fabd (DEFAULT_TABLE table=test-table [id=c710287148a44da4a17372afe6644727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:56.486318 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cad0819cdea4d52a73473a6a031fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:56.486601 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 0cad0819cdea4d52a73473a6a031fabd (DEFAULT_TABLE table=test-table [id=c710287148a44da4a17372afe6644727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:21:56.486904 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cad0819cdea4d52a73473a6a031fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:21:56.488819 16215 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap starting.
I20260620 14:21:56.489256 16216 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Bootstrap starting.
I20260620 14:21:56.489483 16215 tablet_bootstrap.cc:654] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:56.490063 16216 tablet_bootstrap.cc:654] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:56.490074 16217 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Bootstrap starting.
I20260620 14:21:56.490093 16215 log.cc:824] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:56.490656 16217 tablet_bootstrap.cc:654] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Neither blocks nor log segments found. Creating new log.
I20260620 14:21:56.490758 16216 log.cc:824] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:56.491143 16217 log.cc:824] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:21:56.491508 16216 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: No bootstrap required, opened a new log
I20260620 14:21:56.491505 16215 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: No bootstrap required, opened a new log
I20260620 14:21:56.491629 16216 ts_tablet_manager.cc:1397] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:56.491637 16215 ts_tablet_manager.cc:1397] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:21:56.491899 16217 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: No bootstrap required, opened a new log
I20260620 14:21:56.491968 16217 ts_tablet_manager.cc:1397] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:21:56.493189 16216 raft_consensus.cc:348] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.493347 16216 raft_consensus.cc:374] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:56.493391 16216 raft_consensus.cc:729] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f07dca7db84b95a05845b800323abc, State: Initialized, Role: FOLLOWER
I20260620 14:21:56.493494 16216 consensus_queue.cc:260] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.493564 16215 raft_consensus.cc:348] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.493757 16215 raft_consensus.cc:374] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:56.493799 16215 raft_consensus.cc:729] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ef2cd13f664f1f8dbc452f42340d60, State: Initialized, Role: FOLLOWER
I20260620 14:21:56.493796 16216 ts_tablet_manager.cc:1428] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:21:56.493912 16215 consensus_queue.cc:260] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.493870 16217 raft_consensus.cc:348] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.494012 16217 raft_consensus.cc:374] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:21:56.494055 16199 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
I20260620 14:21:56.494058 16217 raft_consensus.cc:729] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204fe3cbac9e4497babca71d31acd60f, State: Initialized, Role: FOLLOWER
I20260620 14:21:56.494167 16215 ts_tablet_manager.cc:1428] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:21:56.494287 15931 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
I20260620 14:21:56.494182 16217 consensus_queue.cc:260] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.494685 16217 ts_tablet_manager.cc:1428] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:21:56.494844 16065 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
W20260620 14:21:56.537245 16066 tablet.cc:2369] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:21:56.636169 15932 tablet.cc:2369] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:21:56.687065 16223 raft_consensus.cc:482] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:21:56.687258 16223 raft_consensus.cc:504] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.687714 16223 leader_election.cc:288] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1:45865), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
I20260620 14:21:56.690732 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.687924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45865 (local address 127.0.98.2:53357)
0620 14:21:56.687988 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.688001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.688052 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.688248 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.688249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.688251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:56.688338 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.688340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.689051 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.689053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.689856 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.689859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.690448 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.690449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.690528 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.690532 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.690595 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.690645 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:21:56.690732 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.688055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53357 (local address 127.0.98.1:45865)
0620 14:21:56.688103 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.688105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.688115 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:21:56.688150 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.688151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.688163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.688238 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:56.688422 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.688951 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.689940 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.690365 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.690417 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.690426 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.690607 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.690662 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:21:56.690999 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.688052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37715 (local address 127.0.98.3:37091)
0620 14:21:56.688102 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.688104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.688227 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:21:56.688262 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.688263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.688275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.688336 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:21:56.688550 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.689075 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.690130 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.690599 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.690660 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.690670 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.690880 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.690931 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:21:56.691228 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.687942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37091 (local address 127.0.98.2:37715)
0620 14:21:56.688126 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.688136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:21:56.688203 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:21:56.688347 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:21:56.688348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:21:56.688349 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:21:56.688440 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.688442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.689160 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.689161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.690042 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:21:56.690045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.690675 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.690676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:21:56.690773 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.690777 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:21:56.691113 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 14:21:56.691157 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":15,"thread_start_us":97,"threads_started":1}
I20260620 14:21:56.691200 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cad0819cdea4d52a73473a6a031fabd" candidate_uuid: "204fe3cbac9e4497babca71d31acd60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ef2cd13f664f1f8dbc452f42340d60" is_pre_election: true
I20260620 14:21:56.691377 15885 raft_consensus.cc:2454] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 204fe3cbac9e4497babca71d31acd60f in term 0.
I20260620 14:21:56.691617 15952 leader_election.cc:302] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204fe3cbac9e4497babca71d31acd60f, 54ef2cd13f664f1f8dbc452f42340d60; no voters:
I20260620 14:21:56.691644 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cad0819cdea4d52a73473a6a031fabd" candidate_uuid: "204fe3cbac9e4497babca71d31acd60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f07dca7db84b95a05845b800323abc" is_pre_election: true
I20260620 14:21:56.691762 16223 raft_consensus.cc:2781] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:21:56.691823 16223 raft_consensus.cc:482] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:21:56.691812 16153 raft_consensus.cc:2454] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 204fe3cbac9e4497babca71d31acd60f in term 0.
I20260620 14:21:56.691856 16223 raft_consensus.cc:3037] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:56.692694 16223 raft_consensus.cc:504] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.692852 16223 leader_election.cc:288] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [CANDIDATE]: Term 1 election: Requested vote from peers 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1:45865), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
I20260620 14:21:56.693013 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cad0819cdea4d52a73473a6a031fabd" candidate_uuid: "204fe3cbac9e4497babca71d31acd60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ef2cd13f664f1f8dbc452f42340d60"
I20260620 14:21:56.693135 15885 raft_consensus.cc:3037] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:56.693161 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cad0819cdea4d52a73473a6a031fabd" candidate_uuid: "204fe3cbac9e4497babca71d31acd60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f07dca7db84b95a05845b800323abc"
I20260620 14:21:56.693229 16153 raft_consensus.cc:3037] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:21:56.694115 16153 raft_consensus.cc:2454] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 204fe3cbac9e4497babca71d31acd60f in term 1.
I20260620 14:21:56.694115 15885 raft_consensus.cc:2454] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 204fe3cbac9e4497babca71d31acd60f in term 1.
I20260620 14:21:56.694305 15955 leader_election.cc:302] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204fe3cbac9e4497babca71d31acd60f, 70f07dca7db84b95a05845b800323abc; no voters:
I20260620 14:21:56.694393 16223 raft_consensus.cc:2781] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:21:56.694535 16223 raft_consensus.cc:686] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 1 LEADER]: Becoming Leader. State: Replica: 204fe3cbac9e4497babca71d31acd60f, State: Running, Role: LEADER
I20260620 14:21:56.694633 16223 consensus_queue.cc:237] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:21:56.695531 15738 catalog_manager.cc:5495] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f reported cstate change: term changed from 0 to 1, leader changed from <none> to 204fe3cbac9e4497babca71d31acd60f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "204fe3cbac9e4497babca71d31acd60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } health_report { overall_health: HEALTHY } } }
W20260620 14:21:56.697492 16200 tablet.cc:2369] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:21:56.713915 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.711465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.0.98.2:40733)
0620 14:21:56.711532 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:21:56.711534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:21:56.711543 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:21:56.711603 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:21:56.711604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:21:56.711625 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:21:56.711718 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:21:56.711860 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.712328 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.713117 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:21:56.713318 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:21:56.713404 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:21:56.713586 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:21:56.713684 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:21:56.713780 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:21:56.713799 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:21:56.713820 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:21:56.715898 15885 raft_consensus.cc:1261] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Refusing update from remote peer 204fe3cbac9e4497babca71d31acd60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:56.715898 16153 raft_consensus.cc:1261] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Refusing update from remote peer 204fe3cbac9e4497babca71d31acd60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:21:56.716382 16225 consensus_queue.cc:1035] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:56.716553 16223 consensus_queue.cc:1035] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:21:56.719051 16229 mvcc.cc:205] Tried to move back new op lower bound from 7298929937265004544 to 7298929937181655040. Current Snapshot: MvccSnapshot[applied={T|T < 7298929937265004544}]
I20260620 14:21:56.719910 16230 mvcc.cc:205] Tried to move back new op lower bound from 7298929937265004544 to 7298929937181655040. Current Snapshot: MvccSnapshot[applied={T|T < 7298929937265004544 or (T in {7298929937265004544})}]
I20260620 14:21:56.721171 15738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:56.721273 15738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:21:56.722013 15738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Sending DeleteTablet for 3 replicas of tablet 0cad0819cdea4d52a73473a6a031fabd
I20260620 14:21:56.722348 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cad0819cdea4d52a73473a6a031fabd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55708
I20260620 14:21:56.722370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 15802
I20260620 14:21:56.722611 16238 tablet_replica.cc:330] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: stopping tablet replica
I20260620 14:21:56.722716 16238 raft_consensus.cc:2229] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:21:56.722801 16238 raft_consensus.cc:2258] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:56.723255 16238 ts_tablet_manager.cc:1905] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:56.723335 16133 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cad0819cdea4d52a73473a6a031fabd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45486
I20260620 14:21:56.723531 16239 tablet_replica.cc:330] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: stopping tablet replica
I20260620 14:21:56.723678 16239 raft_consensus.cc:2229] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:21:56.723816 16239 raft_consensus.cc:2258] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:21:56.724257 16239 ts_tablet_manager.cc:1905] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:21:56.725275 16238 ts_tablet_manager.cc:1918] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:56.725353 16238 log.cc:1196] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/0cad0819cdea4d52a73473a6a031fabd
I20260620 14:21:56.725629 16238 ts_tablet_manager.cc:1939] T 0cad0819cdea4d52a73473a6a031fabd P 204fe3cbac9e4497babca71d31acd60f: Deleting consensus metadata
I20260620 14:21:56.726006 15725 catalog_manager.cc:4841] TS 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733): tablet 0cad0819cdea4d52a73473a6a031fabd (table test-table [id=c710287148a44da4a17372afe6644727]) successfully deleted
I20260620 14:21:56.726739 16239 ts_tablet_manager.cc:1918] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:21:56.726876 16239 log.cc:1196] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/0cad0819cdea4d52a73473a6a031fabd
I20260620 14:21:56.727214 16239 ts_tablet_manager.cc:1939] T 0cad0819cdea4d52a73473a6a031fabd P 70f07dca7db84b95a05845b800323abc: Deleting consensus metadata
I20260620 14:21:56.727670 15727 catalog_manager.cc:4841] TS 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091): tablet 0cad0819cdea4d52a73473a6a031fabd (table test-table [id=c710287148a44da4a17372afe6644727]) successfully deleted
W20260620 14:21:56.729800 15724 connection.cc:535] client connection to 127.0.98.1:45865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:21:56.729905 15724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:21:56.730162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 15936
I20260620 14:21:56.730340 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:21:56.730217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45865 (local address 127.0.0.1:35924)
0620 14:21:56.730267 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:21:56.730303 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:21:56.731084 15724 catalog_manager.cc:4584] TS 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1:45865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0cad0819cdea4d52a73473a6a031fabd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45865: connect: Connection refused (error 111)
I20260620 14:21:56.734575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 16070
I20260620 14:21:56.739105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 15708
I20260620 14:22:16.744491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--webserver_interface=127.0.98.62
--webserver_port=40349
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:16.824632 16241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:16.824843 16241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:16.824867 16241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:16.826242 16241 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:22:16.826293 16241 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:22:16.826308 16241 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:22:16.826319 16241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:16.826339 16241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:16.826388 16241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:16.826406 16241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:16.826428 16241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:16.828019 16241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:16.828276 16241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:16.828562 16241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:16.837453 16247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:16.837453 16250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:16.837484 16241 server_base.cc:1034] running on GCE node
W20260620 14:22:16.837535 16248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:16.838110 16241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:16.838451 16241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:16.839634 16241 hybrid_clock.cc:648] HybridClock initialized: now 1781965336839621 us; error 31 us; skew 500 ppm
Jun 20 14:22:16 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:16.849372 16241 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:16.849870 16241 webserver.cc:460] Webserver started at http://127.0.98.62:40349/ using document root <none> and password file <none>
I20260620 14:22:16.850118 16241 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:16.850210 16241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:16.851683 16241 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:22:16.852383 16257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:16.852592 16241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:16.852710 16241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "65f7daed94244fc8ba5ebfee80763c14"
format_stamp: "Formatted at 2026-06-20 14:21:55 on dist-test-slave-h6n5"
server_key: "f7f25515efca2410f23cf6903de1f9ed"
server_key_iv: "7f7f9c8526b272ed01181c2c5cc72bde"
server_key_version: "encryptionkey@0"
I20260620 14:22:16.853123 16241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:16.867676 16241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:16.870147 16241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:16.870314 16241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:16.874620 16241 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40047
I20260620 14:22:16.874702 16309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40047 every 8 connection(s)
I20260620 14:22:16.875115 16241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:16.876897 16310 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:22:16.877547 16310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Bootstrap starting.
I20260620 14:22:16.879745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 16241
I20260620 14:22:16.880038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45865
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40047
--webserver_port=33145
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:22:16.887176 16310 log.cc:824] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:16.890028 16310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:22:16.890359 16310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Bootstrap complete.
I20260620 14:22:16.893975 16310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:16.894280 16310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7daed94244fc8ba5ebfee80763c14, State: Initialized, Role: FOLLOWER
I20260620 14:22:16.894445 16310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:16.894510 16310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:16.894560 16310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:16.894604 16310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:22:16.895934 16310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:16.896045 16310 leader_election.cc:302] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65f7daed94244fc8ba5ebfee80763c14; no voters:
I20260620 14:22:16.896200 16310 leader_election.cc:288] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:22:16.896279 16314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:22:16.896443 16310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:16.896488 16314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 LEADER]: Becoming Leader. State: Replica: 65f7daed94244fc8ba5ebfee80763c14, State: Running, Role: LEADER
I20260620 14:22:16.896597 16314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:16.897048 16316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65f7daed94244fc8ba5ebfee80763c14. Latest consensus state: current_term: 2 leader_uuid: "65f7daed94244fc8ba5ebfee80763c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
I20260620 14:22:16.896917 16315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "65f7daed94244fc8ba5ebfee80763c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
I20260620 14:22:16.897217 16316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:16.897219 16315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:16.898347 16330 catalog_manager.cc:1259] Loaded cluster ID: de793ee03bb942a79c213df2a3ea1789
I20260620 14:22:16.898399 16330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: loading cluster ID for follower catalog manager: success
I20260620 14:22:16.899642 16330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: acquiring CA information for follower catalog manager: success
I20260620 14:22:16.900050 16330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:22:16.900127 16323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:16.900446 16323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:16.900503 16323 catalog_manager.cc:1259] Loaded cluster ID: de793ee03bb942a79c213df2a3ea1789
I20260620 14:22:16.900568 16323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:16.900715 16323 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:16.900813 16323 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Loaded TSK: 0
I20260620 14:22:16.906047 16323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Generated new TSK 1
I20260620 14:22:16.906131 16323 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:22:16.985948 16312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:16.986181 16312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:16.986223 16312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:16.987478 16312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:16.987567 16312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:16.987617 16312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:16.989158 16312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33145
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:16.989387 16312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:16.989637 16312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:16.994747 16337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:16.994825 16340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:16.995012 16338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:16.995182 16312 server_base.cc:1034] running on GCE node
I20260620 14:22:16.995393 16312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:16.995699 16312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:16.996814 16312 hybrid_clock.cc:648] HybridClock initialized: now 1781965336996799 us; error 30 us; skew 500 ppm
Jun 20 14:22:16 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:16.999590 16312 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:22:17.000118 16312 webserver.cc:460] Webserver started at http://127.0.98.1:33145/ using document root <none> and password file <none>
I20260620 14:22:17.000356 16312 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:17.000437 16312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:17.001875 16312 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:17.002631 16347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.002882 16312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:17.002983 16312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "54ef2cd13f664f1f8dbc452f42340d60"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "b235985c3cdbcc57702e6eb68914538d"
server_key_iv: "722a8d0b9acfe0b82993cfc1da046b0e"
server_key_version: "encryptionkey@0"
I20260620 14:22:17.003430 16312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:17.020246 16312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:17.022796 16312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:17.022976 16312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:17.023247 16312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:17.023823 16354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:22:17.025358 16312 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:22:17.025424 16312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:22:17.025470 16312 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:22:17.026324 16312 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:22:17.026372 16312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:22:17.026480 16354 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap starting.
I20260620 14:22:17.034772 16312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45865
I20260620 14:22:17.034998 16461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45865 every 8 connection(s)
I20260620 14:22:17.035212 16312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:17.037431 16354 log.cc:824] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:17.037446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 16312
I20260620 14:22:17.037758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40733
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40047
--webserver_port=38907
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:22:17.041209 16354 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:22:17.041519 16354 tablet_bootstrap.cc:492] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap complete.
I20260620 14:22:17.042717 16354 ts_tablet_manager.cc:1397] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20260620 14:22:17.044504 16354 raft_consensus.cc:348] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:22:17.044894 16354 raft_consensus.cc:729] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ef2cd13f664f1f8dbc452f42340d60, State: Initialized, Role: FOLLOWER
I20260620 14:22:17.045079 16354 consensus_queue.cc:260] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } }
I20260620 14:22:17.045372 16354 ts_tablet_manager.cc:1428] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
Jun 20 14:22:17 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:17.050431 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.035883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:36305)
0620 14:22:17.036113 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.036165 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.036812 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.038198 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.038203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.038616 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:17.038896 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.038903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.040922 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.040923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.042344 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.042347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.043315 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.043317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.043481 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.044127 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:17.044136 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:17.045979 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:17.048514 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.048521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.048524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.048871 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.049156 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.049157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.049158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.049320 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.049747 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:17.049753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:17.049931 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.050065 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.050229 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":131,"threads_started":1}
I20260620 14:22:17.050755 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.036014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36305 (local address 127.0.98.62:40047)
0620 14:22:17.037726 (+ 1712us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.037735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.037775 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:22:17.037918 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.037925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.037982 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.039254 (+ 1272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:17.039365 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.040777 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.042463 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.043173 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.043811 (+ 638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.046141 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:17.046145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:17.046146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:17.046158 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:17.048416 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.048965 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.048968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.048969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.049078 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.049395 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.049396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.049396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.049669 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:17.049745 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.050163 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.050292 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":70,"threads_started":1}
I20260620 14:22:17.051589 16462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260620 14:22:17.051678 16462 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:17.051836 16462 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260620 14:22:17.052639 16274 ts_manager.cc:194] Registered new tserver with Master: 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1:45865)
I20260620 14:22:17.053814 16274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36305
I20260620 14:22:17.054715 16462 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
Jun 20 14:22:17 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:22:17.063632 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.053193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45865 (local address 127.0.0.1:56888)
0620 14:22:17.053563 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.053597 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.053654 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.053930 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.053934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.055061 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:17.055203 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.055205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.056435 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.056436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.057433 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.057436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.058087 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.058091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.058178 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.058767 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:17.058773 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:17.060425 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:17.062433 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.062440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.062442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.062687 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.062900 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.062901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.062901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.062966 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.063256 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:17.063263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:17.063315 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.063399 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.063511 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":83,"threads_started":1}
I20260620 14:22:17.063900 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.053422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.0.98.1:45865)
0620 14:22:17.053764 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.053769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.053777 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:22:17.053806 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.053808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.053821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.053876 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:17.055258 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.056331 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.057516 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.058000 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.058091 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.060513 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:17.060520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:17.060520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:17.060527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:17.062341 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.062773 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.062776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.062777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.062836 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.063029 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.063029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.063030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.063190 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:17.063265 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.063448 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.063543 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":63,"threads_started":1}
I20260620 14:22:17.064414 16396 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cad0819cdea4d52a73473a6a031fabd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:21:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56888
I20260620 14:22:17.065026 16471 tablet_replica.cc:330] stopping tablet replica
I20260620 14:22:17.065129 16471 raft_consensus.cc:2229] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:17.065196 16471 raft_consensus.cc:2258] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:17.065523 16471 ts_tablet_manager.cc:1905] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:17.067729 16471 ts_tablet_manager.cc:1918] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:17.067802 16471 log.cc:1196] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal/wals/0cad0819cdea4d52a73473a6a031fabd
I20260620 14:22:17.068034 16471 ts_tablet_manager.cc:1939] T 0cad0819cdea4d52a73473a6a031fabd P 54ef2cd13f664f1f8dbc452f42340d60: Deleting consensus metadata
I20260620 14:22:17.068585 16258 catalog_manager.cc:4841] TS 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1:45865): tablet 0cad0819cdea4d52a73473a6a031fabd (table test-table [id=c710287148a44da4a17372afe6644727]) successfully deleted
W20260620 14:22:17.126355 16467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:17.126541 16467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:17.126564 16467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:17.127737 16467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:17.127776 16467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:17.127802 16467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:17.129230 16467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38907
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:17.129464 16467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:17.129696 16467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:17.134723 16477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:17.134806 16467 server_base.cc:1034] running on GCE node
W20260620 14:22:17.134711 16476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:17.134711 16479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:17.135138 16467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:17.135331 16467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:17.136463 16467 hybrid_clock.cc:648] HybridClock initialized: now 1781965337136449 us; error 30 us; skew 500 ppm
Jun 20 14:22:17 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:17.138878 16467 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:22:17.139290 16467 webserver.cc:460] Webserver started at http://127.0.98.2:38907/ using document root <none> and password file <none>
I20260620 14:22:17.139516 16467 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:17.139621 16467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:17.140901 16467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:17.141566 16486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.141781 16467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:17.141887 16467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "204fe3cbac9e4497babca71d31acd60f"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "095a950d882dc7e40f418d5fc3232b78"
server_key_iv: "2b28306b3ec64320ab18e1922d707d59"
server_key_version: "encryptionkey@0"
I20260620 14:22:17.142217 16467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:17.164383 16467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:17.166451 16467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:17.166636 16467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:17.166870 16467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:17.167191 16467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:17.167244 16467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.167284 16467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:17.167309 16467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.173420 16467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40733
I20260620 14:22:17.173480 16599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40733 every 8 connection(s)
I20260620 14:22:17.173789 16467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:22:17.174135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 16467
I20260620 14:22:17.174420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37091
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40047
--webserver_port=42185
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:22:17 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:17.184614 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.174629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46263 (local address 127.0.98.62:40047)
0620 14:22:17.174674 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.174676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.174884 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:22:17.175416 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.175418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.175430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.175483 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:17.176448 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.177249 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.178411 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.178798 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.178880 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.181361 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:17.181363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:17.181363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:17.181372 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:17.183118 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.183515 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.183516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.183517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.183612 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.183877 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.183878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.183879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.184036 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:17.184115 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.184365 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.184462 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:22:17.184634 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.174581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:46263)
0620 14:22:17.174763 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.174799 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.175310 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.175590 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.175594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.176164 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:17.176344 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.176349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.177359 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.177361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.178318 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.178321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.178892 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.178893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.179006 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.179660 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:17.179669 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:17.181266 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:17.183208 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.183214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.183215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.183437 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.183693 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.183694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.183694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.183800 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.184108 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:17.184113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:17.184225 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.184315 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.184436 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260620 14:22:17.185146 16600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260620 14:22:17.185256 16600 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:17.185451 16600 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260620 14:22:17.185832 16274 ts_manager.cc:194] Registered new tserver with Master: 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733)
I20260620 14:22:17.186203 16274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46263
W20260620 14:22:17.256407 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:17.256619 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:17.256655 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:17.257784 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:17.257826 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:17.257853 16603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:17.259303 16603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42185
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:17.259529 16603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:17.259783 16603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:17.264602 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:17.264662 16609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:17.264592 16608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:17.264961 16603 server_base.cc:1034] running on GCE node
I20260620 14:22:17.265193 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:17.265429 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:17.266609 16603 hybrid_clock.cc:648] HybridClock initialized: now 1781965337266566 us; error 75 us; skew 500 ppm
Jun 20 14:22:17 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:17.269197 16603 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:22:17.269629 16603 webserver.cc:460] Webserver started at http://127.0.98.3:42185/ using document root <none> and password file <none>
I20260620 14:22:17.269858 16603 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:17.269935 16603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:17.271221 16603 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:17.271817 16618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.271968 16603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.272054 16603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "70f07dca7db84b95a05845b800323abc"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "c61c1a8baaf27601c32440316cf1d931"
server_key_iv: "b5314b06d05881512c12925dfcd44dae"
server_key_version: "encryptionkey@0"
I20260620 14:22:17.272390 16603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:17.287539 16603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:17.289880 16603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:17.290031 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:17.290254 16603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:17.290542 16603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:17.290629 16603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.290673 16603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:17.290714 16603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:17.296594 16603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37091
I20260620 14:22:17.296648 16731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37091 every 8 connection(s)
I20260620 14:22:17.297030 16603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:17.299516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 16603
Jun 20 14:22:17 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:17.307466 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.299956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.0.98.62:40047)
0620 14:22:17.300090 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.300093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.300216 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:22:17.300292 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.300293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.300344 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.300430 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:17.301053 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.301546 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.302358 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.302574 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.302659 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.304427 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:17.304428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:17.304429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:17.304436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:17.306156 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.306592 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.306593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.306593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.306676 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.306906 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.306907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.306907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.307012 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:17.307154 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.307216 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.307343 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260620 14:22:17.307476 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.297774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38567 (local address 127.0.98.62:40047)
0620 14:22:17.297842 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.297844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.298108 (+ 264us) server_negotiation.cc:407] Connection header received
0620 14:22:17.298754 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.298755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.298780 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.298833 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:17.299892 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.300450 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.301471 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.301683 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.301770 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.304010 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:17.304011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:17.304012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:17.304019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:17.305818 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.306223 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.306224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.306225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.306283 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.306592 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.306593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.306593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.306750 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:17.306808 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.307217 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.307396 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":42}
I20260620 14:22:17.307806 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.297774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:38567)
0620 14:22:17.297970 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.298005 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.298625 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.298913 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.298917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.299486 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:17.299793 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.299798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.300554 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.300555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.301370 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.301372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.301781 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.301782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.301923 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.302471 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:17.302479 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:17.303913 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:17.305912 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.305918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.305920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.306144 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.306361 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:17.306362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:17.306369 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:17.306505 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:17.306819 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:17.306826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:17.307018 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.307196 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.307366 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260620 14:22:17.308202 16732 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40047
I20260620 14:22:17.308296 16732 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:17.308434 16732 heartbeater.cc:511] Master 127.0.98.62:40047 requested a full tablet report, sending...
I20260620 14:22:17.308844 16273 ts_manager.cc:194] Registered new tserver with Master: 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
I20260620 14:22:17.309369 16273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38567
I20260620 14:22:17.319999 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:17.322472 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.320373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.0.98.62:40047)
0620 14:22:17.320442 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.320444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.320482 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:22:17.320569 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.320570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.320601 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.320659 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:17.320759 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.321208 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.321905 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.322079 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.322128 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.322257 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:17.322298 (+ 41us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:22:17.322402 (+ 104us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57262: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 14:22:17.322527 16465 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57262: token expired
W20260620 14:22:17.322613 16204 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40047)
I20260620 14:22:17.328526 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.322872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.0.98.62:40047)
0620 14:22:17.322930 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.322931 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.322941 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:22:17.322984 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.322984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.322993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.323053 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:17.323516 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.323999 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.324703 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.324895 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.324975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.326075 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:17.326076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:17.326077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:17.326084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:17.327335 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.327765 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.327766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.327766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.327827 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:17.328024 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:17.328025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:17.328026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:17.328142 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:17.328243 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.328355 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.328428 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:22:17.329692 16273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:22:17.331324 16273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:22:17.333434 16396 tablet_service.cc:1467] Processing CreateTablet for tablet d87f15b5d1db497d811843d23ac91ac6 (DEFAULT_TABLE table=test-table [id=df4b1ba60034461f95ff27fa7922f13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:17.333642 16396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87f15b5d1db497d811843d23ac91ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:17.335000 16354 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap starting.
I20260620 14:22:17.335340 16354 tablet_bootstrap.cc:654] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:17.335927 16354 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: No bootstrap required, opened a new log
I20260620 14:22:17.335981 16354 ts_tablet_manager.cc:1397] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:17.336315 16354 raft_consensus.cc:348] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.336390 16354 raft_consensus.cc:374] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:17.336436 16354 raft_consensus.cc:729] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ef2cd13f664f1f8dbc452f42340d60, State: Initialized, Role: FOLLOWER
I20260620 14:22:17.336488 16354 consensus_queue.cc:260] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.336563 16354 ts_tablet_manager.cc:1428] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:17.337735 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.333186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40733 (local address 127.0.0.1:54130)
0620 14:22:17.333302 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.333314 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.333354 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.333772 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.333773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.333775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:17.333823 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.333825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.335618 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.335619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.336584 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.336587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.337506 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.337506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.337577 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.337595 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.337656 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.337688 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 14:22:17.337756 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.333293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.0.98.2:40733)
0620 14:22:17.333611 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.333616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.333627 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:17.333667 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.333669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.333683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.333759 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:17.333909 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.335157 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.336680 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.337434 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.337498 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.337545 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.337668 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.337696 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":83,"threads_started":1}
I20260620 14:22:17.338049 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.333191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37091 (local address 127.0.0.1:48738)
0620 14:22:17.333541 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.333551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.333599 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.334094 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.334095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.334097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:17.334136 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.334138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.335410 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.335411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.336596 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.336599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.337838 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.337839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.337904 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.337907 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.337955 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.337988 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":213,"thread_start_us":89,"threads_started":1}
I20260620 14:22:17.338091 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.333299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.0.98.3:37091)
0620 14:22:17.333925 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.333929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.333940 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:17.333979 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.333981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.333996 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.334082 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:17.334211 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.335303 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.336698 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.337434 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.337502 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.337558 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.337972 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.338011 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":101,"threads_started":1}
I20260620 14:22:17.338413 16534 tablet_service.cc:1467] Processing CreateTablet for tablet d87f15b5d1db497d811843d23ac91ac6 (DEFAULT_TABLE table=test-table [id=df4b1ba60034461f95ff27fa7922f13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:17.338658 16666 tablet_service.cc:1467] Processing CreateTablet for tablet d87f15b5d1db497d811843d23ac91ac6 (DEFAULT_TABLE table=test-table [id=df4b1ba60034461f95ff27fa7922f13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:17.338761 16534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87f15b5d1db497d811843d23ac91ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:17.338958 16666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87f15b5d1db497d811843d23ac91ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:17.342344 16743 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Bootstrap starting.
I20260620 14:22:17.342953 16743 tablet_bootstrap.cc:654] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:17.343385 16743 log.cc:824] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:17.343711 16742 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Bootstrap starting.
I20260620 14:22:17.344089 16743 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: No bootstrap required, opened a new log
I20260620 14:22:17.344141 16743 ts_tablet_manager.cc:1397] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:22:17.344308 16742 tablet_bootstrap.cc:654] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:17.344841 16742 log.cc:824] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:17.345541 16742 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: No bootstrap required, opened a new log
I20260620 14:22:17.345602 16742 ts_tablet_manager.cc:1397] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:22:17.345996 16743 raft_consensus.cc:348] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.346191 16743 raft_consensus.cc:374] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:17.346239 16743 raft_consensus.cc:729] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204fe3cbac9e4497babca71d31acd60f, State: Initialized, Role: FOLLOWER
I20260620 14:22:17.346351 16743 consensus_queue.cc:260] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.346639 16743 ts_tablet_manager.cc:1428] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:22:17.346659 16600 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
I20260620 14:22:17.347724 16742 raft_consensus.cc:348] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.347860 16742 raft_consensus.cc:374] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:17.347913 16742 raft_consensus.cc:729] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f07dca7db84b95a05845b800323abc, State: Initialized, Role: FOLLOWER
I20260620 14:22:17.348021 16742 consensus_queue.cc:260] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.348308 16742 ts_tablet_manager.cc:1428] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:17.348332 16732 heartbeater.cc:503] Master 127.0.98.62:40047 was elected leader, sending a full tablet report...
I20260620 14:22:17.392421 16468 raft_consensus.cc:482] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:17.392612 16468 raft_consensus.cc:504] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.392973 16468 leader_election.cc:288] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
I20260620 14:22:17.396078 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.393134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40733 (local address 127.0.98.1:39305)
0620 14:22:17.393212 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.393226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.393280 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.393572 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.393573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.393575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:17.393634 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.393636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.394337 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.394338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.395115 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.395117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.395751 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.395752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.395817 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.395840 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.395892 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.395951 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":3}
I20260620 14:22:17.396070 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.393211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39305 (local address 127.0.98.2:40733)
0620 14:22:17.393256 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.393258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.393269 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:17.393381 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.393382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.393395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.393453 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:17.393727 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.394259 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.395213 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.395677 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.395734 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.395743 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.395911 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.395954 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:22:17.396080 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.393222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52917 (local address 127.0.98.3:37091)
0620 14:22:17.393266 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.393268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.393575 (+ 307us) server_negotiation.cc:407] Connection header received
0620 14:22:17.393604 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.393605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.393617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.393676 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:17.393868 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.394333 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.395269 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.395699 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.395793 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.395803 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.395947 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.396006 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:22:17.396306 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.393143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37091 (local address 127.0.98.1:52917)
0620 14:22:17.393353 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.393364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:17.393430 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:17.393727 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:17.393727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:17.393729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:17.393799 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.393800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.394411 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.394412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.395183 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:17.395186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.395804 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.395804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:17.395858 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.395861 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:17.395935 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:22:17.395970 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":12,"thread_start_us":98,"threads_started":1}
I20260620 14:22:17.396581 16554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87f15b5d1db497d811843d23ac91ac6" candidate_uuid: "54ef2cd13f664f1f8dbc452f42340d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204fe3cbac9e4497babca71d31acd60f" is_pre_election: true
I20260620 14:22:17.396739 16554 raft_consensus.cc:2454] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54ef2cd13f664f1f8dbc452f42340d60 in term 0.
I20260620 14:22:17.396706 16686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87f15b5d1db497d811843d23ac91ac6" candidate_uuid: "54ef2cd13f664f1f8dbc452f42340d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f07dca7db84b95a05845b800323abc" is_pre_election: true
I20260620 14:22:17.396868 16686 raft_consensus.cc:2454] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54ef2cd13f664f1f8dbc452f42340d60 in term 0.
I20260620 14:22:17.396944 16349 leader_election.cc:302] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 204fe3cbac9e4497babca71d31acd60f, 54ef2cd13f664f1f8dbc452f42340d60; no voters:
I20260620 14:22:17.397116 16468 raft_consensus.cc:2781] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:17.397209 16468 raft_consensus.cc:482] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:17.397245 16468 raft_consensus.cc:3037] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:17.398010 16468 raft_consensus.cc:504] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.398157 16468 leader_election.cc:288] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 1 election: Requested vote from peers 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
I20260620 14:22:17.398314 16686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87f15b5d1db497d811843d23ac91ac6" candidate_uuid: "54ef2cd13f664f1f8dbc452f42340d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f07dca7db84b95a05845b800323abc"
I20260620 14:22:17.398325 16554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87f15b5d1db497d811843d23ac91ac6" candidate_uuid: "54ef2cd13f664f1f8dbc452f42340d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204fe3cbac9e4497babca71d31acd60f"
I20260620 14:22:17.398392 16686 raft_consensus.cc:3037] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:17.398401 16554 raft_consensus.cc:3037] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:17.399067 16686 raft_consensus.cc:2454] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54ef2cd13f664f1f8dbc452f42340d60 in term 1.
I20260620 14:22:17.399108 16554 raft_consensus.cc:2454] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54ef2cd13f664f1f8dbc452f42340d60 in term 1.
I20260620 14:22:17.399227 16351 leader_election.cc:302] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54ef2cd13f664f1f8dbc452f42340d60, 70f07dca7db84b95a05845b800323abc; no voters:
I20260620 14:22:17.399323 16468 raft_consensus.cc:2781] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:17.399492 16468 raft_consensus.cc:686] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 LEADER]: Becoming Leader. State: Replica: 54ef2cd13f664f1f8dbc452f42340d60, State: Running, Role: LEADER
I20260620 14:22:17.399595 16468 consensus_queue.cc:237] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:17.400313 16273 catalog_manager.cc:5495] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "54ef2cd13f664f1f8dbc452f42340d60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } health_report { overall_health: HEALTHY } } }
I20260620 14:22:17.410480 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.408056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.0.98.1:45865)
0620 14:22:17.408095 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:17.408097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:17.408105 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:22:17.408174 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:17.408175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:17.408191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:17.408293 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:17.408403 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.409014 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.409701 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:17.409919 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:17.409995 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:17.410158 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:17.410258 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:22:17.410331 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:17.410386 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:22:17.410408 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:22:17.411780 16686 raft_consensus.cc:1261] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Refusing update from remote peer 54ef2cd13f664f1f8dbc452f42340d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:17.411780 16554 raft_consensus.cc:1261] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 1 FOLLOWER]: Refusing update from remote peer 54ef2cd13f664f1f8dbc452f42340d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:17.412102 16468 consensus_queue.cc:1035] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:17.412256 16749 consensus_queue.cc:1035] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:17.414513 16752 mvcc.cc:205] Tried to move back new op lower bound from 7298930022036353024 to 7298930021989351424. Current Snapshot: MvccSnapshot[applied={T|T < 7298930022036353024}]
I20260620 14:22:17.415750 16754 mvcc.cc:205] Tried to move back new op lower bound from 7298930022036353024 to 7298930021989351424. Current Snapshot: MvccSnapshot[applied={T|T < 7298930022036353024 or (T in {7298930022036353024})}]
I20260620 14:22:17.416690 16274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:17.416783 16274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:17.417524 16274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Sending DeleteTablet for 3 replicas of tablet d87f15b5d1db497d811843d23ac91ac6
I20260620 14:22:17.417799 16396 tablet_service.cc:1514] Processing DeleteTablet for tablet d87f15b5d1db497d811843d23ac91ac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56888
I20260620 14:22:17.417855 16666 tablet_service.cc:1514] Processing DeleteTablet for tablet d87f15b5d1db497d811843d23ac91ac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48738
I20260620 14:22:17.417889 16471 tablet_replica.cc:330] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: stopping tablet replica
I20260620 14:22:17.417932 16471 raft_consensus.cc:2229] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:22:17.417927 16534 tablet_service.cc:1514] Processing DeleteTablet for tablet d87f15b5d1db497d811843d23ac91ac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54130
I20260620 14:22:17.418017 16471 raft_consensus.cc:2258] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:17.418195 16761 tablet_replica.cc:330] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: stopping tablet replica
I20260620 14:22:17.418190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 16312
I20260620 14:22:17.418351 16471 ts_tablet_manager.cc:1905] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:17.418285 16761 raft_consensus.cc:2229] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:17.418756 16761 raft_consensus.cc:2258] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:17.418757 16762 tablet_replica.cc:330] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: stopping tablet replica
I20260620 14:22:17.418826 16762 raft_consensus.cc:2229] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:17.418890 16762 raft_consensus.cc:2258] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:17.419188 16761 ts_tablet_manager.cc:1905] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:17.419250 16762 ts_tablet_manager.cc:1905] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:17.421195 16761 ts_tablet_manager.cc:1918] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:17.421216 16762 ts_tablet_manager.cc:1918] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:17.421250 16761 log.cc:1196] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/d87f15b5d1db497d811843d23ac91ac6
I20260620 14:22:17.421279 16762 log.cc:1196] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/d87f15b5d1db497d811843d23ac91ac6
I20260620 14:22:17.421478 16761 ts_tablet_manager.cc:1939] T d87f15b5d1db497d811843d23ac91ac6 P 70f07dca7db84b95a05845b800323abc: Deleting consensus metadata
I20260620 14:22:17.421607 16762 ts_tablet_manager.cc:1939] T d87f15b5d1db497d811843d23ac91ac6 P 204fe3cbac9e4497babca71d31acd60f: Deleting consensus metadata
I20260620 14:22:17.421855 16261 catalog_manager.cc:4841] TS 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091): tablet d87f15b5d1db497d811843d23ac91ac6 (table test-table [id=df4b1ba60034461f95ff27fa7922f13c]) successfully deleted
I20260620 14:22:17.421953 16259 catalog_manager.cc:4841] TS 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733): tablet d87f15b5d1db497d811843d23ac91ac6 (table test-table [id=df4b1ba60034461f95ff27fa7922f13c]) successfully deleted
W20260620 14:22:17.424396 16258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45865 (error 108)
I20260620 14:22:17.424664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 16467
I20260620 14:22:17.425369 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:17.424773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45865 (local address 127.0.0.1:56918)
0620 14:22:17.425275 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:17.425328 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478}
W20260620 14:22:17.427677 16258 catalog_manager.cc:4584] TS 54ef2cd13f664f1f8dbc452f42340d60 (127.0.98.1:45865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d87f15b5d1db497d811843d23ac91ac6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45865: connect: Connection refused (error 111)
I20260620 14:22:17.429481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 16603
I20260620 14:22:17.433871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 16241
I20260620 14:22:37.440281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--webserver_interface=127.0.98.62
--webserver_port=40349
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:37.521770 16764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:37.521986 16764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:37.522009 16764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:37.523191 16764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:22:37.523231 16764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:22:37.523244 16764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:22:37.523257 16764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:37.523269 16764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:37.523317 16764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:37.523357 16764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:37.523373 16764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:37.524883 16764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:37.525139 16764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:37.525375 16764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:37.530630 16772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:37.530637 16769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:37.530750 16764 server_base.cc:1034] running on GCE node
W20260620 14:22:37.530649 16770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:37.531111 16764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:37.531369 16764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:37.532545 16764 hybrid_clock.cc:648] HybridClock initialized: now 1781965357532517 us; error 50 us; skew 500 ppm
Jun 20 14:22:37 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:37.538647 16764 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:37.539150 16764 webserver.cc:460] Webserver started at http://127.0.98.62:40349/ using document root <none> and password file <none>
I20260620 14:22:37.539381 16764 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:37.539459 16764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:37.540874 16764 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:37.541556 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:37.541769 16764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:37.541862 16764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "65f7daed94244fc8ba5ebfee80763c14"
format_stamp: "Formatted at 2026-06-20 14:21:55 on dist-test-slave-h6n5"
server_key: "f7f25515efca2410f23cf6903de1f9ed"
server_key_iv: "7f7f9c8526b272ed01181c2c5cc72bde"
server_key_version: "encryptionkey@0"
I20260620 14:22:37.542309 16764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:37.552929 16764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:37.555159 16764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:37.555327 16764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:37.559147 16764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40047
I20260620 14:22:37.559201 16831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40047 every 8 connection(s)
I20260620 14:22:37.559613 16764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:37.561065 16832 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:22:37.561730 16832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Bootstrap starting.
I20260620 14:22:37.565544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 16764
I20260620 14:22:37.565856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45865
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40047
--webserver_port=33145
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:22:37.570216 16832 log.cc:824] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:37.572486 16832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:22:37.572786 16832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Bootstrap complete.
I20260620 14:22:37.574976 16832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:37.575224 16832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f7daed94244fc8ba5ebfee80763c14, State: Initialized, Role: FOLLOWER
I20260620 14:22:37.575414 16832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:37.575501 16832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:37.575578 16832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:37.575632 16832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:22:37.576598 16832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:37.576721 16832 leader_election.cc:302] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65f7daed94244fc8ba5ebfee80763c14; no voters:
I20260620 14:22:37.576930 16832 leader_election.cc:288] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:22:37.576964 16836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:22:37.577077 16836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [term 3 LEADER]: Becoming Leader. State: Replica: 65f7daed94244fc8ba5ebfee80763c14, State: Running, Role: LEADER
I20260620 14:22:37.577195 16832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:37.577170 16836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } }
I20260620 14:22:37.577461 16836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65f7daed94244fc8ba5ebfee80763c14. Latest consensus state: current_term: 3 leader_uuid: "65f7daed94244fc8ba5ebfee80763c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
I20260620 14:22:37.577538 16836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:37.577466 16837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "65f7daed94244fc8ba5ebfee80763c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f7daed94244fc8ba5ebfee80763c14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40047 } } }
I20260620 14:22:37.577711 16837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:37.577751 16840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:37.578612 16840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:37.578733 16840 catalog_manager.cc:1259] Loaded cluster ID: de793ee03bb942a79c213df2a3ea1789
I20260620 14:22:37.578776 16840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:37.579838 16840 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:37.580040 16840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Loaded TSK: 0
I20260620 14:22:37.580101 16840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65f7daed94244fc8ba5ebfee80763c14: Loaded TSK: 1
F20260620 14:22:37.580318 16840 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc6cfd0f9ad google::LogMessage::Fail() at ??:0
@ 0x7fc6cfd1288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc6cfd0f4de google::LogMessage::Flush() at ??:0
@ 0x7fc6cfd10a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc6d247c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc6d2479d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc6d7c691cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc6d7c6948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc6d7c69594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc6d7c2fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc6d7c39ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc6d1bdd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc6d1bd2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc6d2a366db start_thread at ??:0
@ 0x7fc6ce80d61f clone at ??:0
W20260620 14:22:37.687165 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:37.687462 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:37.687501 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:37.689357 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:37.689435 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:37.689481 16834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:37.691769 16834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33145
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:37.692039 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:37.692293 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:37.699209 16876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:37.699224 16873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:37.699714 16834 server_base.cc:1034] running on GCE node
W20260620 14:22:37.699787 16874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:37.700047 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:37.700344 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:37.701527 16834 hybrid_clock.cc:648] HybridClock initialized: now 1781965357701505 us; error 36 us; skew 500 ppm
Jun 20 14:22:37 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:37.705037 16834 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:22:37.705655 16834 webserver.cc:460] Webserver started at http://127.0.98.1:33145/ using document root <none> and password file <none>
I20260620 14:22:37.705950 16834 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:37.706027 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:37.707938 16834 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:22:37.708727 16883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:37.708961 16834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:37.709059 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "54ef2cd13f664f1f8dbc452f42340d60"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "b235985c3cdbcc57702e6eb68914538d"
server_key_iv: "722a8d0b9acfe0b82993cfc1da046b0e"
server_key_version: "encryptionkey@0"
I20260620 14:22:37.709432 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:37.717856 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:37.720091 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:37.720249 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:37.720482 16834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:37.720927 16890 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:22:37.722944 16834 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:22:37.723023 16834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:22:37.723070 16834 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:22:37.723781 16834 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:22:37.723834 16834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:37.723905 16890 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap starting.
I20260620 14:22:37.725854 16890 log.cc:824] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:37.729113 16890 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:22:37.729432 16890 tablet_bootstrap.cc:492] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Bootstrap complete.
I20260620 14:22:37.729591 16890 ts_tablet_manager.cc:1397] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:22:37.730906 16834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45865
I20260620 14:22:37.731227 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:37.731526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 16834
I20260620 14:22:37.731418 16890 raft_consensus.cc:348] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:37.731777 16890 raft_consensus.cc:729] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ef2cd13f664f1f8dbc452f42340d60, State: Initialized, Role: FOLLOWER
I20260620 14:22:37.731837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40733
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40047
--webserver_port=38907
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:22:37.731941 16890 consensus_queue.cc:260] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:37.732201 16890 ts_tablet_manager.cc:1428] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:37.732492 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45865 every 8 connection(s)
W20260620 14:22:37.800062 17003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:22:37.733085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:43133)
0620 14:22:37.734277 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:37.734323 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:37.734498 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:37.799840 (+ 65342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":87,"threads_started":1}
W20260620 14:22:37.800334 16999 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40047 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40047: Client connection negotiation failed: client connection to 127.0.98.62:40047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:22:37.800696 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:37.800582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:60669)
0620 14:22:37.800638 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:37.800676 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:22:37.801086 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:37.800996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:37947)
0620 14:22:37.801047 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:37.801070 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:22:37.801193 16999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:22:37.815526 17002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:37.815809 17002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:37.815846 17002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:37.817008 17002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:37.817075 17002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:37.817121 17002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:37.818600 17002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38907
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:37.818871 17002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:37.819110 17002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:37.824127 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:37.824134 17011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:37.824152 17009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:37.824429 17002 server_base.cc:1034] running on GCE node
I20260620 14:22:37.824666 17002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:37.824913 17002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:37.826076 17002 hybrid_clock.cc:648] HybridClock initialized: now 1781965357826060 us; error 27 us; skew 500 ppm
Jun 20 14:22:37 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:37.828719 17002 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:22:37.829164 17002 webserver.cc:460] Webserver started at http://127.0.98.2:38907/ using document root <none> and password file <none>
I20260620 14:22:37.829404 17002 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:37.829474 17002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:37.830804 17002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:37.831399 17018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:37.831630 17002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:22:37.831720 17002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "204fe3cbac9e4497babca71d31acd60f"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "095a950d882dc7e40f418d5fc3232b78"
server_key_iv: "2b28306b3ec64320ab18e1922d707d59"
server_key_version: "encryptionkey@0"
I20260620 14:22:37.832091 17002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:37.855324 17002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:37.857282 17002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:37.857450 17002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:37.857717 17002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:37.858028 17002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:37.858081 17002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:37.858122 17002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:37.858151 17002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:37.864177 17002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40733
I20260620 14:22:37.864223 17131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40733 every 8 connection(s)
I20260620 14:22:37.864540 17002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:22:37.865478 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:37.865131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:60581)
0620 14:22:37.865275 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:37.865342 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20260620 14:22:37.865877 17132 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40047 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40047: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
I20260620 14:22:37.866161 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:37.866073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:35363)
0620 14:22:37.866117 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:37.866143 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:22:37.866425 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:37.866352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:48363)
0620 14:22:37.866388 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:37.866411 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:22:37.866539 17132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:22:37.870970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17002
I20260620 14:22:37.871233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37091
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40047
--webserver_port=42185
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:37.955245 17135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:37.955477 17135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:37.955514 17135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:37.956688 17135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:37.956765 17135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:37.956809 17135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:37.958261 17135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42185
--tserver_master_addrs=127.0.98.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:37.958525 17135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:37.958755 17135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:37.963917 17140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:37.963888 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:37.963936 17135 server_base.cc:1034] running on GCE node
W20260620 14:22:37.963903 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:37.964299 17135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:37.964562 17135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:37.965744 17135 hybrid_clock.cc:648] HybridClock initialized: now 1781965357965738 us; error 37 us; skew 500 ppm
Jun 20 14:22:37 dist-test-slave-h6n5 krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:37.968472 17135 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:22:37.968974 17135 webserver.cc:460] Webserver started at http://127.0.98.3:42185/ using document root <none> and password file <none>
I20260620 14:22:37.969158 17135 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:37.969200 17135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:37.970495 17135 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:37.971246 17150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:37.971652 17135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:37.971735 17135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "70f07dca7db84b95a05845b800323abc"
format_stamp: "Formatted at 2026-06-20 14:21:56 on dist-test-slave-h6n5"
server_key: "c61c1a8baaf27601c32440316cf1d931"
server_key_iv: "b5314b06d05881512c12925dfcd44dae"
server_key_version: "encryptionkey@0"
I20260620 14:22:37.972121 17135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:37.992843 17135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:37.995349 17135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:37.995476 17135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:37.995740 17135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:37.996008 17135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:37.996040 17135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:37.996062 17135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:37.996075 17135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:38.002085 17135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37091
I20260620 14:22:38.002130 17263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37091 every 8 connection(s)
I20260620 14:22:38.002476 17135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:38.003654 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:38.003142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:45221)
0620 14:22:38.003355 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:38.003434 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
W20260620 14:22:38.004066 17264 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40047 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40047: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
I20260620 14:22:38.004364 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:38.004273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:58219)
0620 14:22:38.004309 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:38.004341 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 14:22:38.004703 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:38.004606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:50587)
0620 14:22:38.004655 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:38.004680 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:22:38.004843 17264 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:22:38.006181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17135
I20260620 14:22:38.801842 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:38.801552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:33185)
0620 14:22:38.801702 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:38.801752 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260620 14:22:38.867038 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:38.866825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:35489)
0620 14:22:38.866941 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:38.866976 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260620 14:22:39.005398 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:39.005181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:60165)
0620 14:22:39.005304 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:39.005339 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260620 14:22:39.297590 17271 raft_consensus.cc:482] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:39.297705 17271 raft_consensus.cc:504] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:39.297982 17271 leader_election.cc:288] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
Jun 20 14:22:39 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 14:22:39 dist-test-slave-h6n5 krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 14:22:39.308557 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:39.298338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36855 (local address 127.0.98.2:40733)
0620 14:22:39.298504 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:39.298512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:39.298540 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:22:39.298986 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:39.298988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:39.299028 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:39.299170 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:39.300093 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.300988 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.301910 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.302313 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.302397 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:39.304788 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:39.304792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:39.304792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:39.304802 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:39.306779 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:39.307258 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:39.307262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:39.307262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:39.307379 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:39.307682 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:39.307684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:39.307684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:39.307926 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:39.308015 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:39.308311 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:22:39.308447 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260620 14:22:39.308553 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:39.298140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40733 (local address 127.0.98.1:36855)
0620 14:22:39.298197 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:39.298209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:39.298277 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:39.299214 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:39.299217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:39.299883 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:39.299986 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:39.299988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.301087 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.301088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:39.301826 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:39.301829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.302399 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.302400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:39.302537 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:39.303188 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:39.303192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:39.304685 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:39.306874 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:39.306881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:39.306883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:39.307178 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:39.307451 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:39.307452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:39.307452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:39.307613 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:39.308008 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:39.308015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:39.308145 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:22:39.308244 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:22:39.308439 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":3}
I20260620 14:22:39.308760 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:39.298140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37091 (local address 127.0.98.1:33727)
0620 14:22:39.298261 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:39.298272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:39.298325 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:39.299215 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:39.299217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:39.299536 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:39.299793 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:39.299799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.300951 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.300952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:39.301825 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:39.301827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.302449 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.302450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:39.302536 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:39.303130 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:39.303138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:39.305150 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:39.307109 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:39.307110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:39.307111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:39.307343 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:39.307647 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:39.307648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:39.307648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:39.307711 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:39.308090 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:39.308091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:39.308157 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 14:22:39.308246 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:22:39.308523 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":191,"thread_start_us":58,"threads_started":1}
I20260620 14:22:39.308576 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:39.298314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33727 (local address 127.0.98.3:37091)
0620 14:22:39.298502 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:39.298512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:39.298540 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:22:39.298982 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:39.298984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:39.299028 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:39.299183 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:39.299918 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.300846 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.301960 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:39.302363 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:39.302477 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:39.305268 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:39.305272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:39.305272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:39.305281 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:39.307020 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:39.307441 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:39.307444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:39.307445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:39.307542 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:39.307777 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:39.307778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:39.307778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:39.308014 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:39.308108 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:39.308308 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:22:39.308458 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
W20260620 14:22:39.309523 16885 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
W20260620 14:22:39.309916 16887 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
I20260620 14:22:39.309957 16887 leader_election.cc:302] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54ef2cd13f664f1f8dbc452f42340d60; no voters: 204fe3cbac9e4497babca71d31acd60f, 70f07dca7db84b95a05845b800323abc
I20260620 14:22:39.310050 17271 raft_consensus.cc:2726] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:22:39.802467 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:39.802291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:58373)
0620 14:22:39.802370 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:39.802428 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:22:39.867666 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:39.867455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:49051)
0620 14:22:39.867592 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:39.867628 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
I20260620 14:22:40.006094 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:40.005867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:55725)
0620 14:22:40.006011 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:40.006049 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260620 14:22:40.803248 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:40.803044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:33489)
0620 14:22:40.803166 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:40.803201 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260620 14:22:40.868330 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:40.868109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:56859)
0620 14:22:40.868232 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:40.868275 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260620 14:22:41.006938 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:41.006713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:56351)
0620 14:22:41.006856 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:41.006894 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260620 14:22:41.357061 17280 raft_consensus.cc:482] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:41.357177 17280 raft_consensus.cc:504] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:41.357519 17280 leader_election.cc:288] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
W20260620 14:22:41.357928 16887 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
W20260620 14:22:41.357996 16885 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
I20260620 14:22:41.358023 16887 leader_election.cc:302] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54ef2cd13f664f1f8dbc452f42340d60; no voters: 204fe3cbac9e4497babca71d31acd60f, 70f07dca7db84b95a05845b800323abc
I20260620 14:22:41.358107 17280 raft_consensus.cc:2726] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:22:41.803910 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:41.803690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:37749)
0620 14:22:41.803823 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:41.803862 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260620 14:22:41.868999 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:41.868784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:39189)
0620 14:22:41.868915 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:41.868955 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260620 14:22:42.007741 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:42.007465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:38333)
0620 14:22:42.007660 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:42.007698 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":62,"threads_started":1}
I20260620 14:22:42.804672 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:42.804405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:37933)
0620 14:22:42.804581 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:42.804624 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260620 14:22:42.869714 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:42.869505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:51991)
0620 14:22:42.869626 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:42.869669 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260620 14:22:43.008455 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:43.008228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:58277)
0620 14:22:43.008369 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:43.008408 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260620 14:22:43.656422 17287 raft_consensus.cc:482] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:43.656518 17287 raft_consensus.cc:504] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:43.656829 17287 leader_election.cc:288] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
W20260620 14:22:43.657264 16887 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
W20260620 14:22:43.657344 16885 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
I20260620 14:22:43.657384 16885 leader_election.cc:302] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54ef2cd13f664f1f8dbc452f42340d60; no voters: 204fe3cbac9e4497babca71d31acd60f, 70f07dca7db84b95a05845b800323abc
I20260620 14:22:43.657466 17287 raft_consensus.cc:2726] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:22:43.805460 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:43.805198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:60281)
0620 14:22:43.805359 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:43.805404 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":69,"threads_started":1}
I20260620 14:22:43.870447 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:43.870211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:55209)
0620 14:22:43.870358 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:43.870403 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260620 14:22:44.009299 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:44.009032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:50609)
0620 14:22:44.009201 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:44.009247 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260620 14:22:44.806262 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:44.806012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:47965)
0620 14:22:44.806172 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:44.806213 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260620 14:22:44.871258 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:44.871034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:46351)
0620 14:22:44.871179 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:44.871213 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260620 14:22:45.010106 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:45.009827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:39419)
0620 14:22:45.010020 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:45.010060 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260620 14:22:45.807096 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:45.806814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:56683)
0620 14:22:45.807001 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:45.807050 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
I20260620 14:22:45.872071 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:45.871805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:37263)
0620 14:22:45.871985 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:45.872026 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20260620 14:22:46.010800 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:46.010567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:46427)
0620 14:22:46.010714 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:46.010755 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260620 14:22:46.807915 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:46.807695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:50195)
0620 14:22:46.807828 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:46.807870 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260620 14:22:46.872818 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:46.872606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:36885)
0620 14:22:46.872728 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:46.872772 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260620 14:22:47.011615 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:47.011320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:45027)
0620 14:22:47.011469 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:47.011529 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260620 14:22:47.808624 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:47.808388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:41383)
0620 14:22:47.808545 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:47.808578 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 14:22:47.873509 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:47.873287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:56121)
0620 14:22:47.873424 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:47.873465 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260620 14:22:48.012444 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:48.012263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:47583)
0620 14:22:48.012370 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:48.012401 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260620 14:22:48.684399 17303 raft_consensus.cc:482] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:48.684556 17303 raft_consensus.cc:504] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "204fe3cbac9e4497babca71d31acd60f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "70f07dca7db84b95a05845b800323abc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37091 } } peers { permanent_uuid: "54ef2cd13f664f1f8dbc452f42340d60" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45865 } }
I20260620 14:22:48.684759 17303 leader_election.cc:288] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733), 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091)
W20260620 14:22:48.685243 16885 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 204fe3cbac9e4497babca71d31acd60f (127.0.98.2:40733): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
W20260620 14:22:48.685333 16887 leader_election.cc:341] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70f07dca7db84b95a05845b800323abc (127.0.98.3:37091): Not found: Tablet not found: d87f15b5d1db497d811843d23ac91ac6
I20260620 14:22:48.685395 16887 leader_election.cc:302] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54ef2cd13f664f1f8dbc452f42340d60; no voters: 204fe3cbac9e4497babca71d31acd60f, 70f07dca7db84b95a05845b800323abc
I20260620 14:22:48.685480 17303 raft_consensus.cc:2726] T d87f15b5d1db497d811843d23ac91ac6 P 54ef2cd13f664f1f8dbc452f42340d60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:22:48.809438 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:48.809180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:33809)
0620 14:22:48.809345 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:48.809390 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260620 14:22:48.874313 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:48.874059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:49607)
0620 14:22:48.874223 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:48.874268 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260620 14:22:49.013327 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:49.013012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:35999)
0620 14:22:49.013188 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:49.013244 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260620 14:22:49.810258 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:49.810031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:49107)
0620 14:22:49.810170 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:49.810209 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260620 14:22:49.875015 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:49.874801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:46373)
0620 14:22:49.874925 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:49.874970 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260620 14:22:50.014045 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:50.013831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:54991)
0620 14:22:50.013951 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:50.013997 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260620 14:22:50.811020 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:50.810797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:51327)
0620 14:22:50.810934 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:50.810973 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260620 14:22:50.875901 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:50.875659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:33547)
0620 14:22:50.875813 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:50.875854 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260620 14:22:51.014770 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:51.014556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:42051)
0620 14:22:51.014682 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:51.014722 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260620 14:22:51.811931 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:51.811618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:51535)
0620 14:22:51.811840 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:51.811882 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
I20260620 14:22:51.876817 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:51.876568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:36907)
0620 14:22:51.876723 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:51.876768 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260620 14:22:52.015476 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:52.015243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:41091)
0620 14:22:52.015388 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:52.015426 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260620 14:22:52.812770 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:52.812548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.1:60061)
0620 14:22:52.812692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:52.812726 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260620 14:22:52.877619 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:52.877355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.2:46625)
0620 14:22:52.877492 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:52.877550 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260620 14:22:53.014286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 16834
I20260620 14:22:53.016255 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:53.015999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40047 (local address 127.0.98.3:49491)
0620 14:22:53.016148 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:53.016196 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":51,"threads_started":1}
I20260620 14:22:53.019169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17002
I20260620 14:22:53.024231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17135
2026-06-20T14:22:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58354 ms)
[----------] 30 tests from SecurityITest (233312 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:22:53 dist-test-slave-h6n5 krb5kdc[17321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:22:53 dist-test-slave-h6n5 krb5kdc[17321](info): set up 2 sockets
Jun 20 14:22:53 dist-test-slave-h6n5 krb5kdc[17321](info): commencing operation
krb5kdc: starting...
W20260620 14:22:54.067129 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:22:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:22:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:22:54.187801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:54.269565 17334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:54.269810 17334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:54.269852 17334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:54.271015 17334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:54.271099 17334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:54.271126 17334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:54.271152 17334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:54.271183 17334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:54.272665 17334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:54.272943 17334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:54.273173 17334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:54.277989 17339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.278069 17340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.278241 17334 server_base.cc:1034] running on GCE node
W20260620 14:22:54.278278 17342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.278515 17334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:54.278813 17334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:54.279994 17334 hybrid_clock.cc:648] HybridClock initialized: now 1781965374279984 us; error 46 us; skew 500 ppm
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:54.282692 17334 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:54.283201 17334 webserver.cc:460] Webserver started at http://127.0.98.62:38319/ using document root <none> and password file <none>
I20260620 14:22:54.283414 17334 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:54.283459 17334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:54.283622 17334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:54.284734 17334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "c35ca6221ccc403bbbc8506969a884a6"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "423ce6ac1fe90586d57fd52e5e70ab85"
server_key_iv: "8d244f88dd3513eb513d27832939e5b9"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.285058 17334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "c35ca6221ccc403bbbc8506969a884a6"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "423ce6ac1fe90586d57fd52e5e70ab85"
server_key_iv: "8d244f88dd3513eb513d27832939e5b9"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.286412 17334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:22:54.287169 17349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.287441 17334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:54.287524 17334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "c35ca6221ccc403bbbc8506969a884a6"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "423ce6ac1fe90586d57fd52e5e70ab85"
server_key_iv: "8d244f88dd3513eb513d27832939e5b9"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.287644 17334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.300693 17334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.302814 17334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.302958 17334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.306792 17334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42043
I20260620 14:22:54.306888 17401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42043 every 8 connection(s)
I20260620 14:22:54.307199 17334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:54.308123 17402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:54.310461 17402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6: Bootstrap starting.
I20260620 14:22:54.311058 17402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:54.311353 17402 log.cc:824] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:54.312111 17402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6: No bootstrap required, opened a new log
I20260620 14:22:54.312628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17334
I20260620 14:22:54.312719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:22:54.312954 392 external_mini_cluster.cc:1428] Setting key 6816cc8635c32facff55ff04745a81af
I20260620 14:22:54.314349 17402 raft_consensus.cc:348] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35ca6221ccc403bbbc8506969a884a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260620 14:22:54.314510 17402 raft_consensus.cc:374] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:54.314556 17402 raft_consensus.cc:729] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35ca6221ccc403bbbc8506969a884a6, State: Initialized, Role: FOLLOWER
I20260620 14:22:54.314752 17402 consensus_queue.cc:260] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35ca6221ccc403bbbc8506969a884a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260620 14:22:54.314853 17402 raft_consensus.cc:388] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:54.314899 17402 raft_consensus.cc:482] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:54.314947 17402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:54.315485 17402 raft_consensus.cc:504] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35ca6221ccc403bbbc8506969a884a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260620 14:22:54.315629 17402 leader_election.cc:302] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c35ca6221ccc403bbbc8506969a884a6; no voters:
I20260620 14:22:54.315820 17402 leader_election.cc:288] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:22:54.315855 17407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:54.315973 17407 raft_consensus.cc:686] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [term 1 LEADER]: Becoming Leader. State: Replica: c35ca6221ccc403bbbc8506969a884a6, State: Running, Role: LEADER
I20260620 14:22:54.316056 17407 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35ca6221ccc403bbbc8506969a884a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260620 14:22:54.316087 17402 sys_catalog.cc:564] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:54.316414 17407 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35ca6221ccc403bbbc8506969a884a6. Latest consensus state: current_term: 1 leader_uuid: "c35ca6221ccc403bbbc8506969a884a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35ca6221ccc403bbbc8506969a884a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } } }
I20260620 14:22:54.316428 17408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c35ca6221ccc403bbbc8506969a884a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35ca6221ccc403bbbc8506969a884a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } } }
I20260620 14:22:54.316498 17407 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:54.316524 17408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:54.317384 17417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:54.317888 17417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:54.319459 17417 catalog_manager.cc:1347] Generated new cluster ID: 1c7749949ba644c7b0557d148b908721
I20260620 14:22:54.319509 17417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:54.324729 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.313539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.98.62:42043)
0620 14:22:54.313714 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.313723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.313748 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:22:54.314508 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.314517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.314606 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.314765 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.315467 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.316889 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.318236 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.318852 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.318990 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.321420 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.321424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.321424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.321436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.323198 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.323595 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.323599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.323599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.323695 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.323907 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.323908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.323908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.324140 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.324316 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.324398 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.324548 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260620 14:22:54.331290 17417 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:22:54.331867 17417 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:54.344079 17417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6: Generated new TSK 0
I20260620 14:22:54.344287 17417 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:22:54.375103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:54.457491 17428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:54.457695 17428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:54.457717 17428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:54.458817 17428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:54.458855 17428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:54.458881 17428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:54.460449 17428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:54.460659 17428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:54.460830 17428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:54.466027 17434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.466010 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.466006 17436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.466331 17428 server_base.cc:1034] running on GCE node
I20260620 14:22:54.466527 17428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:54.466764 17428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:54.467954 17428 hybrid_clock.cc:648] HybridClock initialized: now 1781965374467940 us; error 36 us; skew 500 ppm
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:54.470469 17428 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:22:54.470896 17428 webserver.cc:460] Webserver started at http://127.0.98.1:46791/ using document root <none> and password file <none>
I20260620 14:22:54.471158 17428 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:54.471235 17428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:54.471385 17428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:54.472471 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0dfe9344243424182e597b3f8be789f"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "056496484750c6b10c4879ae4af8e5a5"
server_key_iv: "4da41eea290815cae49d1bee0f971105"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.472846 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0dfe9344243424182e597b3f8be789f"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "056496484750c6b10c4879ae4af8e5a5"
server_key_iv: "4da41eea290815cae49d1bee0f971105"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.474192 17428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:22:54.474941 17443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.475122 17428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:22:54.475220 17428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "d0dfe9344243424182e597b3f8be789f"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "056496484750c6b10c4879ae4af8e5a5"
server_key_iv: "4da41eea290815cae49d1bee0f971105"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.475301 17428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.483933 17428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.485975 17428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.486168 17428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.486397 17428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:54.486702 17428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:54.486757 17428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.486797 17428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:54.486824 17428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.493098 17428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37725
I20260620 14:22:54.493173 17556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37725 every 8 connection(s)
I20260620 14:22:54.493489 17428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:54.499804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17428
I20260620 14:22:54.499938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:54.500213 392 external_mini_cluster.cc:1428] Setting key 2f4ebc626d7aec9b2662538460d2cf8f
I20260620 14:22:54.504000 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.494248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55907 (local address 127.0.98.62:42043)
0620 14:22:54.494322 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.494324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.494545 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:22:54.495644 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.495645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.495658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.495734 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.496382 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.496902 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.497942 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.498187 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.498246 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.500760 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.500762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.500762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.500769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.502512 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.502918 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.502919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.502920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.502995 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.503284 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.503285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.503285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.503414 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.503470 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.503850 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.503935 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:22:54.504280 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.494201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42043 (local address 127.0.98.1:55907)
0620 14:22:54.494358 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.494400 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.495412 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.495812 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.495816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.496074 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:54.496280 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.496285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.497004 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.497004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.497854 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.497856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.498273 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.498274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.498372 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.499039 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:54.499047 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:54.500648 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:54.502610 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.502617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.502618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.502846 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.503064 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.503065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.503065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.503212 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.503483 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:54.503490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:54.503685 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.503807 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.503983 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260620 14:22:54.504788 17557 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42043
I20260620 14:22:54.504922 17557 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.505105 17557 heartbeater.cc:511] Master 127.0.98.62:42043 requested a full tablet report, sending...
I20260620 14:22:54.505721 17366 ts_manager.cc:194] Registered new tserver with Master: d0dfe9344243424182e597b3f8be789f (127.0.98.1:37725)
I20260620 14:22:54.506353 17366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55907
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:22:54.527143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:54.608825 17562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:54.609074 17562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:54.609115 17562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:54.610327 17562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:54.610395 17562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:54.610440 17562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:54.611963 17562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:54.612241 17562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:54.612478 17562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:54.617511 17570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.617539 17567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.617543 17568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.617815 17562 server_base.cc:1034] running on GCE node
I20260620 14:22:54.618083 17562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:54.618337 17562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:54.619598 17562 hybrid_clock.cc:648] HybridClock initialized: now 1781965374619519 us; error 38 us; skew 500 ppm
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:54.622053 17562 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:22:54.622483 17562 webserver.cc:460] Webserver started at http://127.0.98.2:43109/ using document root <none> and password file <none>
I20260620 14:22:54.622665 17562 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:54.622704 17562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:54.622792 17562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:54.623852 17562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba7d6dcb0c9045629095a4b27fc16e9d"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "fdd9047bcb5b02fd27a421c6ae98b450"
server_key_iv: "27737eb2aef2028f973bdbc92a7ac718"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.624156 17562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba7d6dcb0c9045629095a4b27fc16e9d"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "fdd9047bcb5b02fd27a421c6ae98b450"
server_key_iv: "27737eb2aef2028f973bdbc92a7ac718"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.625399 17562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:22:54.626055 17577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.626248 17562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:22:54.626331 17562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "ba7d6dcb0c9045629095a4b27fc16e9d"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "fdd9047bcb5b02fd27a421c6ae98b450"
server_key_iv: "27737eb2aef2028f973bdbc92a7ac718"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.626384 17562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.635756 17562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.637915 17562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.638022 17562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.638193 17562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:54.638499 17562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:54.638533 17562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.638578 17562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:54.638604 17562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.644448 17562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38099
I20260620 14:22:54.644526 17690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38099 every 8 connection(s)
I20260620 14:22:54.644819 17562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:54.652091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17562
I20260620 14:22:54.652240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:22:54.652465 392 external_mini_cluster.cc:1428] Setting key d7f32e51e17128d70d8e0bec84b29e7a
I20260620 14:22:54.654219 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.645527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58725 (local address 127.0.98.62:42043)
0620 14:22:54.645595 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.645598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.645820 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:22:54.646708 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.646709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.646719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.646767 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.647334 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.647904 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.648893 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.649072 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.649122 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.651184 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.651186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.651187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.651195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.652819 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.653269 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.653270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.653271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.653331 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.653606 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.653606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.653607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.653703 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.653753 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.654072 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.654155 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:22:54.654495 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.645502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42043 (local address 127.0.98.2:58725)
0620 14:22:54.645688 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.645724 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.646621 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.646826 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.646829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.647082 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:54.647265 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.647270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.647987 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.647988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.648815 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.648817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.649142 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.649143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.649242 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.649698 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:54.649706 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:54.651079 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:54.652891 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.652897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.652899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.653180 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.653392 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.653393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.653393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.653541 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.653764 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:54.653770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:54.653939 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.654050 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.654221 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260620 14:22:54.655108 17691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42043
I20260620 14:22:54.655218 17691 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.655412 17691 heartbeater.cc:511] Master 127.0.98.62:42043 requested a full tablet report, sending...
I20260620 14:22:54.655839 17366 ts_manager.cc:194] Registered new tserver with Master: ba7d6dcb0c9045629095a4b27fc16e9d (127.0.98.2:38099)
I20260620 14:22:54.656234 17366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58725
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:22:54.678190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:54.760025 17696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:54.760243 17696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:54.760268 17696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:54.761468 17696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:54.761512 17696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:54.761539 17696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:54.763051 17696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:54.763301 17696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:54.763533 17696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:54.768826 17704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.768834 17701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:54.768862 17702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:54.769203 17696 server_base.cc:1034] running on GCE node
I20260620 14:22:54.769440 17696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:54.769702 17696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:54.770865 17696 hybrid_clock.cc:648] HybridClock initialized: now 1781965374770868 us; error 44 us; skew 500 ppm
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:54.773375 17696 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:22:54.773826 17696 webserver.cc:460] Webserver started at http://127.0.98.3:44481/ using document root <none> and password file <none>
I20260620 14:22:54.774050 17696 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:54.774119 17696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:54.774271 17696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:54.775323 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "4edaf2d4dfdf4636b8e902c48a55a960"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "399969d9799cec4558d73a31bc831dd4"
server_key_iv: "cb8fd9cf3132df88a8b5f21b756c7447"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.775712 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4edaf2d4dfdf4636b8e902c48a55a960"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "399969d9799cec4558d73a31bc831dd4"
server_key_iv: "cb8fd9cf3132df88a8b5f21b756c7447"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.777020 17696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:54.777747 17711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.778023 17696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:54.778126 17696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "4edaf2d4dfdf4636b8e902c48a55a960"
format_stamp: "Formatted at 2026-06-20 14:22:54 on dist-test-slave-h6n5"
server_key: "399969d9799cec4558d73a31bc831dd4"
server_key_iv: "cb8fd9cf3132df88a8b5f21b756c7447"
server_key_version: "encryptionkey@0"
I20260620 14:22:54.778203 17696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:54.792176 17696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:54.794255 17696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:54.794411 17696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:54.794641 17696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:54.794955 17696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:54.795017 17696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.795058 17696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:54.795086 17696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:54.801605 17696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34015
I20260620 14:22:54.801651 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34015 every 8 connection(s)
I20260620 14:22:54.801977 17696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:54.803015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17696
I20260620 14:22:54.803136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:22:54.803346 392 external_mini_cluster.cc:1428] Setting key 13b343f353b6c66f72fd101b96a937fe
Jun 20 14:22:54 dist-test-slave-h6n5 krb5kdc[17321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:54.811688 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.802770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58083 (local address 127.0.98.62:42043)
0620 14:22:54.802842 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.802844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.803114 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:22:54.804410 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.804411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.804421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.804457 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.805054 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.805552 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.806573 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.806762 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.806812 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.809033 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.809034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.809034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.809041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.810380 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.810802 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.810803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.810804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.810855 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.811099 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.811100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.811100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.811198 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.811270 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.811503 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.811613 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:22:54.811924 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.802750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42043 (local address 127.0.98.3:58083)
0620 14:22:54.802924 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.802979 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.804328 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.804532 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.804536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.804794 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:54.804981 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.804985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.805647 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.805648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.806497 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.806499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.806833 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.806834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.806933 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.807434 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:54.807443 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:54.808944 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:54.810462 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.810468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.810470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.810731 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.810934 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:54.810935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:54.810936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:54.811041 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:54.811279 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:54.811283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:54.811396 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.811478 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.811665 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260620 14:22:54.812440 17825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42043
I20260620 14:22:54.812557 17825 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:54.812741 17825 heartbeater.cc:511] Master 127.0.98.62:42043 requested a full tablet report, sending...
I20260620 14:22:54.813107 17366 ts_manager.cc:194] Registered new tserver with Master: 4edaf2d4dfdf4636b8e902c48a55a960 (127.0.98.3:34015)
I20260620 14:22:54.813483 17366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58083
I20260620 14:22:54.814936 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:54.822481 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.816364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.0.98.62:42043)
0620 14:22:54.816420 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.816422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.816600 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:22:54.816673 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.816674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.816684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.816745 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:54.817360 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.817922 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.818687 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.818891 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.818964 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.820138 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:54.820140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:54.820140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:54.820147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:54.821305 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.821649 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.821650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.821650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.821719 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:54.821913 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:54.821914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:54.821914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:54.822067 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:54.822173 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.822305 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.822384 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:22:54.824110 17366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:22:54.825670 17366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:22:54.832121 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.827990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37725 (local address 127.0.0.1:53788)
0620 14:22:54.828168 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.828179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.828243 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.828616 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.828619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.828621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:54.828694 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.828696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.829718 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.829719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.830917 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.830919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.831807 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.831808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.831896 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.831912 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.831980 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.832034 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":47,"threads_started":1}
I20260620 14:22:54.832261 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.827832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38099 (local address 127.0.0.1:39380)
0620 14:22:54.828170 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.828178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.828213 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.828637 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.828638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.828640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:54.828717 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.828719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.830079 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.830080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.831096 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.831099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.831996 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.831997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.832064 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.832068 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.832128 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.832169 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":65,"threads_started":1}
I20260620 14:22:54.832424 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.827773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34015 (local address 127.0.0.1:51254)
0620 14:22:54.827905 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:54.827950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:54.828016 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:54.829016 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:54.829016 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:54.829019 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:54.829058 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.829060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.830308 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.830309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.831318 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:54.831321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.832237 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.832238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:54.832298 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.832302 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:54.832332 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:22:54.832366 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260620 14:22:54.832470 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.828165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.0.98.1:37725)
0620 14:22:54.828341 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.828345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.828353 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:22:54.828385 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.828386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.828399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.828485 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:54.828779 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.829611 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.831005 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.831722 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.831796 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.831854 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.832333 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.832376 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260620 14:22:54.832780 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.827958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.0.98.3:34015)
0620 14:22:54.828852 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.828857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.828867 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:22:54.828904 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.828906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.828921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.829126 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:54.829202 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.830213 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.831402 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.832148 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.832534 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.832599 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.832629 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.832661 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":83,"threads_started":1}
I20260620 14:22:54.832939 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:54.827993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.98.2:38099)
0620 14:22:54.828359 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:54.828405 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:54.828416 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:54.828456 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:54.828458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:54.828472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:54.828552 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:54.828800 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.829945 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.831183 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:54.831896 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:54.831984 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:54.832039 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:54.832159 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 14:22:54.832849 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":76,"threads_started":1}
I20260620 14:22:54.833200 17491 tablet_service.cc:1467] Processing CreateTablet for tablet aa52f97fe9654deaa74a935d6a3a6fe7 (DEFAULT_TABLE table=test-table [id=16d1526d89ac4dbc82745f3726187120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:54.833307 17759 tablet_service.cc:1467] Processing CreateTablet for tablet aa52f97fe9654deaa74a935d6a3a6fe7 (DEFAULT_TABLE table=test-table [id=16d1526d89ac4dbc82745f3726187120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:54.833578 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa52f97fe9654deaa74a935d6a3a6fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:54.833578 17491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa52f97fe9654deaa74a935d6a3a6fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:54.833580 17625 tablet_service.cc:1467] Processing CreateTablet for tablet aa52f97fe9654deaa74a935d6a3a6fe7 (DEFAULT_TABLE table=test-table [id=16d1526d89ac4dbc82745f3726187120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:54.833900 17625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa52f97fe9654deaa74a935d6a3a6fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:54.836525 17839 tablet_bootstrap.cc:492] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: Bootstrap starting.
I20260620 14:22:54.837025 17840 tablet_bootstrap.cc:492] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: Bootstrap starting.
I20260620 14:22:54.837265 17839 tablet_bootstrap.cc:654] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:54.837702 17840 tablet_bootstrap.cc:654] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:54.837917 17839 log.cc:824] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:54.838248 17840 log.cc:824] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:54.838526 17841 tablet_bootstrap.cc:492] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Bootstrap starting.
I20260620 14:22:54.838752 17839 tablet_bootstrap.cc:492] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: No bootstrap required, opened a new log
I20260620 14:22:54.838831 17839 ts_tablet_manager.cc:1397] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:22:54.839150 17841 tablet_bootstrap.cc:654] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:54.839728 17841 log.cc:824] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:54.840140 17840 tablet_bootstrap.cc:492] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: No bootstrap required, opened a new log
I20260620 14:22:54.840246 17840 ts_tablet_manager.cc:1397] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:22:54.840502 17841 tablet_bootstrap.cc:492] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: No bootstrap required, opened a new log
I20260620 14:22:54.840579 17841 ts_tablet_manager.cc:1397] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 14:22:54.841087 17839 raft_consensus.cc:348] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:54.841284 17839 raft_consensus.cc:374] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:54.841322 17839 raft_consensus.cc:729] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7d6dcb0c9045629095a4b27fc16e9d, State: Initialized, Role: FOLLOWER
I20260620 14:22:54.841464 17839 consensus_queue.cc:260] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:54.841733 17839 ts_tablet_manager.cc:1428] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:54.841881 17691 heartbeater.cc:503] Master 127.0.98.62:42043 was elected leader, sending a full tablet report...
I20260620 14:22:54.841909 17840 raft_consensus.cc:348] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:54.842078 17840 raft_consensus.cc:374] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:54.842114 17840 raft_consensus.cc:729] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dfe9344243424182e597b3f8be789f, State: Initialized, Role: FOLLOWER
I20260620 14:22:54.842237 17840 consensus_queue.cc:260] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:54.842511 17557 heartbeater.cc:503] Master 127.0.98.62:42043 was elected leader, sending a full tablet report...
I20260620 14:22:54.842518 17840 ts_tablet_manager.cc:1428] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:22:54.842576 17841 raft_consensus.cc:348] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:54.842760 17841 raft_consensus.cc:374] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:54.842803 17841 raft_consensus.cc:729] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edaf2d4dfdf4636b8e902c48a55a960, State: Initialized, Role: FOLLOWER
I20260620 14:22:54.843000 17841 consensus_queue.cc:260] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:54.843300 17825 heartbeater.cc:503] Master 127.0.98.62:42043 was elected leader, sending a full tablet report...
I20260620 14:22:54.843593 17841 ts_tablet_manager.cc:1428] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260620 14:22:54.895361 17692 tablet.cc:2369] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:22:54.994220 17558 tablet.cc:2369] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:22:55.052580 17826 tablet.cc:2369] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:22:55.115259 17845 raft_consensus.cc:482] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:55.115388 17845 raft_consensus.cc:504] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:55.115918 17845 leader_election.cc:288] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4edaf2d4dfdf4636b8e902c48a55a960 (127.0.98.3:34015), d0dfe9344243424182e597b3f8be789f (127.0.98.1:37725)
I20260620 14:22:55.119874 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.116400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33619 (local address 127.0.98.1:37725)
0620 14:22:55.116450 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.116452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.116514 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:22:55.116587 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.116587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.116596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.116689 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.116862 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.117524 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.118918 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.119465 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.119544 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.119578 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.119763 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.119798 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:22:55.119874 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.116249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57259 (local address 127.0.98.3:34015)
0620 14:22:55.116302 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.116304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.116315 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:55.116356 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.116358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.116368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.116433 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.116659 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.117456 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.118918 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.119499 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.119578 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.119589 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.119762 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.119803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:22:55.119887 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.116132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34015 (local address 127.0.98.2:57259)
0620 14:22:55.116208 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.116236 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.116280 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.116498 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.116499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.116501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.116576 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.116578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.117527 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.117528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.118820 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.118823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.119599 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.119599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.119678 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.119682 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.119741 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.119793 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":13}
I20260620 14:22:55.119894 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.116290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37725 (local address 127.0.98.2:33619)
0620 14:22:55.116460 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.116471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.116526 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.116710 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.116711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.116713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.116766 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.116767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.117594 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.117595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.118821 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.118823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.119594 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.119595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.119686 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.119688 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.119741 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.119793 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260620 14:22:55.120270 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa52f97fe9654deaa74a935d6a3a6fe7" candidate_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" is_pre_election: true
I20260620 14:22:55.120450 17779 raft_consensus.cc:2454] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba7d6dcb0c9045629095a4b27fc16e9d in term 0.
I20260620 14:22:55.120573 17511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa52f97fe9654deaa74a935d6a3a6fe7" candidate_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dfe9344243424182e597b3f8be789f" is_pre_election: true
I20260620 14:22:55.120745 17578 leader_election.cc:302] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edaf2d4dfdf4636b8e902c48a55a960, ba7d6dcb0c9045629095a4b27fc16e9d; no voters:
I20260620 14:22:55.120777 17511 raft_consensus.cc:2454] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba7d6dcb0c9045629095a4b27fc16e9d in term 0.
I20260620 14:22:55.120865 17845 raft_consensus.cc:2781] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:55.120939 17845 raft_consensus.cc:482] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:55.120975 17845 raft_consensus.cc:3037] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:55.121658 17845 raft_consensus.cc:504] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:55.121788 17845 leader_election.cc:288] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [CANDIDATE]: Term 1 election: Requested vote from peers 4edaf2d4dfdf4636b8e902c48a55a960 (127.0.98.3:34015), d0dfe9344243424182e597b3f8be789f (127.0.98.1:37725)
I20260620 14:22:55.121973 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa52f97fe9654deaa74a935d6a3a6fe7" candidate_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edaf2d4dfdf4636b8e902c48a55a960"
I20260620 14:22:55.122033 17511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa52f97fe9654deaa74a935d6a3a6fe7" candidate_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dfe9344243424182e597b3f8be789f"
I20260620 14:22:55.122056 17779 raft_consensus.cc:3037] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:55.122133 17511 raft_consensus.cc:3037] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:55.122784 17779 raft_consensus.cc:2454] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba7d6dcb0c9045629095a4b27fc16e9d in term 1.
I20260620 14:22:55.122808 17511 raft_consensus.cc:2454] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba7d6dcb0c9045629095a4b27fc16e9d in term 1.
I20260620 14:22:55.122962 17578 leader_election.cc:302] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba7d6dcb0c9045629095a4b27fc16e9d, d0dfe9344243424182e597b3f8be789f; no voters:
I20260620 14:22:55.123060 17845 raft_consensus.cc:2781] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:55.123200 17845 raft_consensus.cc:686] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 1 LEADER]: Becoming Leader. State: Replica: ba7d6dcb0c9045629095a4b27fc16e9d, State: Running, Role: LEADER
I20260620 14:22:55.123306 17845 consensus_queue.cc:237] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } }
I20260620 14:22:55.124154 17365 catalog_manager.cc:5495] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d reported cstate change: term changed from 0 to 1, leader changed from <none> to ba7d6dcb0c9045629095a4b27fc16e9d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba7d6dcb0c9045629095a4b27fc16e9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38099 } health_report { overall_health: HEALTHY } } }
I20260620 14:22:55.188378 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.186081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.98.2:38099)
0620 14:22:55.186137 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.186139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.186246 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:22:55.186299 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.186300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.186319 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.186382 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:55.186505 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.186979 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.187719 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.187903 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.187956 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.188081 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:55.188182 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:22:55.188244 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.188283 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.188300 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:22:55.190083 17511 raft_consensus.cc:1261] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 1 FOLLOWER]: Refusing update from remote peer ba7d6dcb0c9045629095a4b27fc16e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:55.190114 17779 raft_consensus.cc:1261] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 1 FOLLOWER]: Refusing update from remote peer ba7d6dcb0c9045629095a4b27fc16e9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:55.190472 17845 consensus_queue.cc:1035] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edaf2d4dfdf4636b8e902c48a55a960" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:55.190619 17849 consensus_queue.cc:1035] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dfe9344243424182e597b3f8be789f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:55.194031 17853 mvcc.cc:205] Tried to move back new op lower bound from 7298930176775835648 to 7298930176505499648. Current Snapshot: MvccSnapshot[applied={T|T < 7298930176775835648}]
I20260620 14:22:55.195305 17365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:55.195393 17365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:55.196183 17365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c35ca6221ccc403bbbc8506969a884a6: Sending DeleteTablet for 3 replicas of tablet aa52f97fe9654deaa74a935d6a3a6fe7
I20260620 14:22:55.196504 17625 tablet_service.cc:1514] Processing DeleteTablet for tablet aa52f97fe9654deaa74a935d6a3a6fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39380
I20260620 14:22:55.196556 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet aa52f97fe9654deaa74a935d6a3a6fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51254
I20260620 14:22:55.196568 17491 tablet_service.cc:1514] Processing DeleteTablet for tablet aa52f97fe9654deaa74a935d6a3a6fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53788
I20260620 14:22:55.196743 17861 tablet_replica.cc:330] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d: stopping tablet replica
I20260620 14:22:55.196835 17861 raft_consensus.cc:2229] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:22:55.196956 17863 tablet_replica.cc:330] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f: stopping tablet replica
I20260620 14:22:55.196970 17862 tablet_replica.cc:330] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: stopping tablet replica
I20260620 14:22:55.197055 17862 raft_consensus.cc:2229] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:55.197054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17428
I20260620 14:22:55.197149 17863 raft_consensus.cc:2229] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:55.197171 17862 raft_consensus.cc:2258] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:55.197320 17863 raft_consensus.cc:2258] T aa52f97fe9654deaa74a935d6a3a6fe7 P d0dfe9344243424182e597b3f8be789f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:55.197764 17862 ts_tablet_manager.cc:1905] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:55.199863 17862 ts_tablet_manager.cc:1918] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:55.199939 17862 log.cc:1196] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/aa52f97fe9654deaa74a935d6a3a6fe7
I20260620 14:22:55.200224 17862 ts_tablet_manager.cc:1939] T aa52f97fe9654deaa74a935d6a3a6fe7 P 4edaf2d4dfdf4636b8e902c48a55a960: Deleting consensus metadata
I20260620 14:22:55.200632 17350 catalog_manager.cc:4841] TS 4edaf2d4dfdf4636b8e902c48a55a960 (127.0.98.3:34015): tablet aa52f97fe9654deaa74a935d6a3a6fe7 (table test-table [id=16d1526d89ac4dbc82745f3726187120]) successfully deleted
W20260620 14:22:55.202836 17350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37725 (error 108)
I20260620 14:22:55.203433 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.203291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37725 (local address 127.0.0.1:53802)
0620 14:22:55.203353 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.203398 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:22:55.203667 17350 catalog_manager.cc:4584] TS d0dfe9344243424182e597b3f8be789f (127.0.98.1:37725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa52f97fe9654deaa74a935d6a3a6fe7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37725: connect: Connection refused (error 111)
I20260620 14:22:55.204524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17562
I20260620 14:22:55.204541 17861 raft_consensus.cc:2258] T aa52f97fe9654deaa74a935d6a3a6fe7 P ba7d6dcb0c9045629095a4b27fc16e9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:55.209879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17696
I20260620 14:22:55.209913 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.209787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38099 (local address 127.0.0.1:39404)
0620 14:22:55.209844 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.209879 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:22:55.214531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17334
2026-06-20T14:22:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2190 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:22:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:22:55Z Disabled control of system clock
I20260620 14:22:55.241559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42765
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:22:55.322956 17868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:55.323169 17868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:55.323189 17868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:55.324271 17868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:55.324306 17868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:55.324319 17868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:55.324332 17868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:55.324344 17868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:55.325583 17868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:55.325758 17868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:55.325973 17868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:55.330657 17876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.330690 17874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.330659 17873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:55.330686 17868 server_base.cc:1034] running on GCE node
I20260620 14:22:55.331111 17868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:55.331391 17868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:55.332587 17868 hybrid_clock.cc:648] HybridClock initialized: now 1781965375332558 us; error 43 us; skew 500 ppm
I20260620 14:22:55.333837 17868 webserver.cc:460] Webserver started at http://127.0.98.62:36277/ using document root <none> and password file <none>
I20260620 14:22:55.334072 17868 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:55.334143 17868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:55.334299 17868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:55.335345 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "41a4942aff2d4f1ab805c53fbb175b56"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "1f83b8729ff2c3c1dd3592531813567e"
server_key_iv: "671421b732630a5e5c8064012944d1b0"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.335717 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "41a4942aff2d4f1ab805c53fbb175b56"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "1f83b8729ff2c3c1dd3592531813567e"
server_key_iv: "671421b732630a5e5c8064012944d1b0"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.336944 17868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:55.337620 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.337787 17868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:22:55.337881 17868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "41a4942aff2d4f1ab805c53fbb175b56"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "1f83b8729ff2c3c1dd3592531813567e"
server_key_iv: "671421b732630a5e5c8064012944d1b0"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.337960 17868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:55.354616 17868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:55.354890 17868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:55.355043 17868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:55.358830 17868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42765
I20260620 14:22:55.358861 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42765 every 8 connection(s)
I20260620 14:22:55.359175 17868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:55.360152 17935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:55.362322 17935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56: Bootstrap starting.
I20260620 14:22:55.362828 17935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:55.363066 17935 log.cc:824] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:55.363739 17935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56: No bootstrap required, opened a new log
I20260620 14:22:55.365247 17935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a4942aff2d4f1ab805c53fbb175b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42765 } }
I20260620 14:22:55.365393 17935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:55.365415 17935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a4942aff2d4f1ab805c53fbb175b56, State: Initialized, Role: FOLLOWER
I20260620 14:22:55.365530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17868
I20260620 14:22:55.365562 17935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a4942aff2d4f1ab805c53fbb175b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42765 } }
I20260620 14:22:55.365649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:22:55.365657 17935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:55.365694 17935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:55.365764 17935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:55.365840 392 external_mini_cluster.cc:1428] Setting key 35a99258b5d8e9ebf71fb87932397c54
I20260620 14:22:55.366510 17935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a4942aff2d4f1ab805c53fbb175b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42765 } }
I20260620 14:22:55.366614 17935 leader_election.cc:302] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a4942aff2d4f1ab805c53fbb175b56; no voters:
I20260620 14:22:55.366756 17935 leader_election.cc:288] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:22:55.366851 17940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:55.366982 17940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [term 1 LEADER]: Becoming Leader. State: Replica: 41a4942aff2d4f1ab805c53fbb175b56, State: Running, Role: LEADER
I20260620 14:22:55.367050 17935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:55.367072 17940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a4942aff2d4f1ab805c53fbb175b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42765 } }
I20260620 14:22:55.367419 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a4942aff2d4f1ab805c53fbb175b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a4942aff2d4f1ab805c53fbb175b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42765 } } }
I20260620 14:22:55.367499 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:55.367681 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a4942aff2d4f1ab805c53fbb175b56. Latest consensus state: current_term: 1 leader_uuid: "41a4942aff2d4f1ab805c53fbb175b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a4942aff2d4f1ab805c53fbb175b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42765 } } }
I20260620 14:22:55.367756 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:55.367884 17946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:55.368360 17946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:55.369971 17946 catalog_manager.cc:1347] Generated new cluster ID: 2757ccff270b426bad8c8b29784988f6
I20260620 14:22:55.370033 17946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:55.372610 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.366335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.98.62:42765)
0620 14:22:55.366580 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.366590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.366624 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:22:55.366755 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.366762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.366837 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.366983 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:55.367316 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.369304 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.370453 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.371046 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.371163 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.371924 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:55.371927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:55.371927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:55.372128 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:55.372167 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:55.372181 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:55.372242 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.372339 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.372409 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260620 14:22:55.391911 17946 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:22:55.392554 17946 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:55.397956 17946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56: Generated new TSK 0
I20260620 14:22:55.398128 17946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:22:55.400882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42765
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:22:55.480612 17959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:55.480841 17959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:55.480875 17959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:55.482049 17959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:55.482086 17959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:55.482154 17959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:55.483513 17959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:55.483799 17959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:55.484094 17959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:55.488998 17964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.489105 17965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.489012 17967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:55.489351 17959 server_base.cc:1034] running on GCE node
I20260620 14:22:55.489547 17959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:55.489790 17959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:55.490954 17959 hybrid_clock.cc:648] HybridClock initialized: now 1781965375490932 us; error 37 us; skew 500 ppm
I20260620 14:22:55.492223 17959 webserver.cc:460] Webserver started at http://127.0.98.1:36271/ using document root <none> and password file <none>
I20260620 14:22:55.492476 17959 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:55.492549 17959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:55.492705 17959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:55.493733 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "2e1eba7717e048dcb0be3d73a0fd128e"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "6ec712b88396afeae89d934a9b711420"
server_key_iv: "42727f65acb5bc95657a529bc8341090"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.494103 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2e1eba7717e048dcb0be3d73a0fd128e"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "6ec712b88396afeae89d934a9b711420"
server_key_iv: "42727f65acb5bc95657a529bc8341090"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.495491 17959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:22:55.496335 17973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.496542 17959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:55.496666 17959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "2e1eba7717e048dcb0be3d73a0fd128e"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "6ec712b88396afeae89d934a9b711420"
server_key_iv: "42727f65acb5bc95657a529bc8341090"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.496757 17959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:55.557394 17959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:55.557757 17959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:55.557927 17959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:55.558175 17959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:55.558507 17959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:55.558570 17959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.558614 17959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:55.558641 17959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.564445 17959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42793
I20260620 14:22:55.564489 18086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42793 every 8 connection(s)
I20260620 14:22:55.564816 17959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:55.566402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 17959
I20260620 14:22:55.566514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:22:55.566720 392 external_mini_cluster.cc:1428] Setting key 44ed3892a9bc85c0c2b7b960b15b3e0a
I20260620 14:22:55.567919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42765
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:22:55.570694 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.565528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41725 (local address 127.0.98.62:42765)
0620 14:22:55.565617 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.565620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.565839 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:22:55.566395 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.566397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.566409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.566487 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:55.567478 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.568142 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.569205 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.569421 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.569509 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.570411 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:55.570412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:55.570412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:55.570418 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:55.570432 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:55.570437 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:55.570485 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.570585 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.570616 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:22:55.570932 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.565512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42765 (local address 127.0.98.1:41725)
0620 14:22:55.565716 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.565751 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.566262 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.566599 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.566604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.566615 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:55.567371 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.567378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.568259 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.568260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.569111 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.569113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.569520 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.569521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.569662 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.570263 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:22:55.570264 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:55.570292 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:22:55.570315 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:55.570484 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:55.570489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:55.570492 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.570588 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.570691 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260620 14:22:55.571458 18087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42765
I20260620 14:22:55.571590 18087 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:55.571784 18087 heartbeater.cc:511] Master 127.0.98.62:42765 requested a full tablet report, sending...
I20260620 14:22:55.572448 17899 ts_manager.cc:194] Registered new tserver with Master: 2e1eba7717e048dcb0be3d73a0fd128e (127.0.98.1:42793)
I20260620 14:22:55.573465 17899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41725
W20260620 14:22:55.649355 18090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:55.649549 18090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:55.649569 18090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:55.650616 18090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:55.650651 18090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:55.650681 18090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:55.652043 18090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:55.652217 18090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:55.652437 18090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:22:55.657379 18090 server_base.cc:1034] running on GCE node
W20260620 14:22:55.657352 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.657352 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.657343 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:55.657824 18090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:55.658069 18090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:55.659207 18090 hybrid_clock.cc:648] HybridClock initialized: now 1781965375659198 us; error 30 us; skew 500 ppm
I20260620 14:22:55.660554 18090 webserver.cc:460] Webserver started at http://127.0.98.2:34805/ using document root <none> and password file <none>
I20260620 14:22:55.660799 18090 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:55.660871 18090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:55.661020 18090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:55.662073 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "3069ba8c11298b8e167ffa2d7685cb9a"
server_key_iv: "67935ea4609c015f4a0906f204ec7271"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.662437 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "3069ba8c11298b8e167ffa2d7685cb9a"
server_key_iv: "67935ea4609c015f4a0906f204ec7271"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.663781 18090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:22:55.664538 18104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.664719 18090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.664812 18090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "3069ba8c11298b8e167ffa2d7685cb9a"
server_key_iv: "67935ea4609c015f4a0906f204ec7271"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.664892 18090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:55.682257 18090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:55.682559 18090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:55.682715 18090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:55.682952 18090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:55.683279 18090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:55.683332 18090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.683367 18090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:55.683395 18090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.689172 18090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41729
I20260620 14:22:55.689251 18217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41729 every 8 connection(s)
I20260620 14:22:55.689568 18090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:22:55.692646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 18090
I20260620 14:22:55.692785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:22:55.692981 392 external_mini_cluster.cc:1428] Setting key 1a4390a63b03a1a43c55d0075cafe1b0
I20260620 14:22:55.694064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42765
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:22:55.695585 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.690359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41027 (local address 127.0.98.62:42765)
0620 14:22:55.690428 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.690430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.690636 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:22:55.691125 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.691126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.691139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.691201 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:55.691991 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.692612 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.693900 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.694284 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.694361 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.695225 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:55.695227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:55.695227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:55.695232 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:55.695251 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:55.695256 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:55.695319 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.695446 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.695483 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:22:55.695844 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.690358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42765 (local address 127.0.98.2:41027)
0620 14:22:55.690528 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.690562 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.691036 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.691274 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.691277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.691286 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:55.691882 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.691888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.692751 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.692752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.693806 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.693808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.694371 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.694372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.694480 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.695081 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:22:55.695082 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:55.695111 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:22:55.695133 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:55.695339 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:55.695345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:55.695348 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.695433 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.695590 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260620 14:22:55.696377 18218 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42765
I20260620 14:22:55.696491 18218 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:55.696676 18218 heartbeater.cc:511] Master 127.0.98.62:42765 requested a full tablet report, sending...
I20260620 14:22:55.697094 17899 ts_manager.cc:194] Registered new tserver with Master: fe751b7e124c42f7ac1df1fe0e0cacd2 (127.0.98.2:41729)
I20260620 14:22:55.697471 17899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41027
W20260620 14:22:55.775877 18221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:55.776126 18221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:55.776163 18221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:55.777346 18221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:55.777415 18221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:55.777456 18221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:55.778798 18221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:55.779062 18221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:55.779371 18221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:55.784267 18226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.784284 18229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:55.784281 18227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:55.784569 18221 server_base.cc:1034] running on GCE node
I20260620 14:22:55.784775 18221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:55.785008 18221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:55.786208 18221 hybrid_clock.cc:648] HybridClock initialized: now 1781965375786185 us; error 39 us; skew 500 ppm
I20260620 14:22:55.787457 18221 webserver.cc:460] Webserver started at http://127.0.98.3:41725/ using document root <none> and password file <none>
I20260620 14:22:55.787752 18221 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:55.787827 18221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:55.787987 18221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:55.789072 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "e45b15473819438f8acb12e36fc780c8"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "8aa2b228a9a930d6cf1bec0937325a92"
server_key_iv: "93bc35267ba5a951d8c8b77cbb7cf1c6"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.789458 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e45b15473819438f8acb12e36fc780c8"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "8aa2b228a9a930d6cf1bec0937325a92"
server_key_iv: "93bc35267ba5a951d8c8b77cbb7cf1c6"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.790791 18221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:22:55.791494 18235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.791679 18221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:22:55.791770 18221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "e45b15473819438f8acb12e36fc780c8"
format_stamp: "Formatted at 2026-06-20 14:22:55 on dist-test-slave-h6n5"
server_key: "8aa2b228a9a930d6cf1bec0937325a92"
server_key_iv: "93bc35267ba5a951d8c8b77cbb7cf1c6"
server_key_version: "encryptionkey@0"
I20260620 14:22:55.791852 18221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:55.816283 18221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:55.816602 18221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:55.816753 18221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:55.816998 18221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:55.817354 18221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:55.817411 18221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.817456 18221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:55.817487 18221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:55.823381 18348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37317 every 8 connection(s)
I20260620 14:22:55.823527 18221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37317
I20260620 14:22:55.823938 18221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:55.829201 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.824768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45949 (local address 127.0.98.62:42765)
0620 14:22:55.824837 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.824839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.825096 (+ 257us) server_negotiation.cc:407] Connection header received
0620 14:22:55.825573 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.825574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.825584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.825654 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:55.826320 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.826812 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.827857 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.828037 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.828091 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.828923 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:55.828924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:55.828924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:55.828928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:55.828941 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:55.828945 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:55.828987 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.829111 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.829133 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:22:55.829455 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.824750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42765 (local address 127.0.98.3:45949)
0620 14:22:55.824933 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.824975 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.825488 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.825706 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.825709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.825717 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:55.826236 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.826240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.826896 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.826897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.827788 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.827790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.828155 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.828156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.828280 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.828792 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:22:55.828792 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:55.828821 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:22:55.828844 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:55.829004 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:55.829041 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:55.829044 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.829099 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.829187 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260620 14:22:55.829690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 18221
I20260620 14:22:55.829788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:22:55.830005 392 external_mini_cluster.cc:1428] Setting key a088980283831afce531c6231d1870b8
I20260620 14:22:55.830017 18349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42765
I20260620 14:22:55.830128 18349 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:55.830339 18349 heartbeater.cc:511] Master 127.0.98.62:42765 requested a full tablet report, sending...
I20260620 14:22:55.830811 17898 ts_manager.cc:194] Registered new tserver with Master: e45b15473819438f8acb12e36fc780c8 (127.0.98.3:37317)
I20260620 14:22:55.831344 17898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45949
I20260620 14:22:55.842097 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:55.846311 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.843534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.98.62:42765)
0620 14:22:55.843629 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.843631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.843757 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:22:55.843798 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.843799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.843808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.843859 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:55.843997 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.844481 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.845226 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.845409 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.845459 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.846118 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:55.846119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:55.846119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:55.846124 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:55.846136 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:55.846139 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:55.846184 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.846232 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.846253 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:22:55.847940 17898 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:22:55.849465 17898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:22:55.855911 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.851595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42793 (local address 127.0.0.1:56966)
0620 14:22:55.851769 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.851808 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.851875 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.852183 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.852186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.852188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.852247 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.852249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.853525 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.853526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.854886 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.854889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.855608 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.855609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.855685 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.855698 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.855767 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.855820 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":81,"thread_start_us":55,"threads_started":1}
I20260620 14:22:55.856019 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.851638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.0.1:36186)
0620 14:22:55.851935 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.851944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.851987 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.852519 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.852520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.852521 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.852556 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.852558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.853813 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.853815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.854749 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.854751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.855786 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.855787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.855849 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.855853 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.855903 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.855946 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":81,"threads_started":1}
I20260620 14:22:55.856105 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.852022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.2:41729)
0620 14:22:55.852314 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.852318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.852327 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:22:55.852362 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.852365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.852378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.852642 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.852710 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.853368 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.854973 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.855707 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.855767 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.855951 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.855978 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.856017 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260620 14:22:55.856252 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.851730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.0.98.3:37317)
0620 14:22:55.852287 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.852292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.852301 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:22:55.852344 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.852346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.852362 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.852455 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.852649 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.853721 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.854834 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.855592 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.855659 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.855749 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.856111 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.856171 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":100,"threads_started":1}
I20260620 14:22:55.856364 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.851759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.98.1:42793)
0620 14:22:55.852007 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.852013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.852023 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:22:55.852061 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.852062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.852075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.852161 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.852330 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.853454 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.854993 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.855474 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.855607 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.855706 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.856227 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.856268 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260620 14:22:55.856660 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.851819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41729 (local address 127.0.0.1:50794)
0620 14:22:55.852099 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.852110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.852162 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.852480 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.852481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.852484 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.852556 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.852560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.853462 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.853463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.854886 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.854889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.855778 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.855779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.855830 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.855833 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.855872 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.855903 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":75,"thread_start_us":62,"threads_started":1}
I20260620 14:22:55.856985 18021 tablet_service.cc:1467] Processing CreateTablet for tablet 14c4495928004c33802a4593867148ec (DEFAULT_TABLE table=test-table [id=0e2c69da5a4d4a64ac574ce27b47ee68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:55.856971 18283 tablet_service.cc:1467] Processing CreateTablet for tablet 14c4495928004c33802a4593867148ec (DEFAULT_TABLE table=test-table [id=0e2c69da5a4d4a64ac574ce27b47ee68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:55.857295 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c4495928004c33802a4593867148ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:55.857318 18283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c4495928004c33802a4593867148ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:55.857331 18152 tablet_service.cc:1467] Processing CreateTablet for tablet 14c4495928004c33802a4593867148ec (DEFAULT_TABLE table=test-table [id=0e2c69da5a4d4a64ac574ce27b47ee68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:55.857621 18152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c4495928004c33802a4593867148ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:55.859912 18363 tablet_bootstrap.cc:492] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e: Bootstrap starting.
I20260620 14:22:55.860589 18363 tablet_bootstrap.cc:654] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:55.860599 18364 tablet_bootstrap.cc:492] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Bootstrap starting.
I20260620 14:22:55.860852 18365 tablet_bootstrap.cc:492] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Bootstrap starting.
I20260620 14:22:55.861133 18363 log.cc:824] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:55.861243 18364 tablet_bootstrap.cc:654] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:55.861490 18365 tablet_bootstrap.cc:654] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:55.861532 18364 log.cc:824] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:55.861819 18365 log.cc:824] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:55.861992 18363 tablet_bootstrap.cc:492] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e: No bootstrap required, opened a new log
I20260620 14:22:55.862068 18363 ts_tablet_manager.cc:1397] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:22:55.862468 18365 tablet_bootstrap.cc:492] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: No bootstrap required, opened a new log
I20260620 14:22:55.862509 18365 ts_tablet_manager.cc:1397] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:22:55.862969 18364 tablet_bootstrap.cc:492] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: No bootstrap required, opened a new log
I20260620 14:22:55.863090 18364 ts_tablet_manager.cc:1397] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:22:55.864440 18365 raft_consensus.cc:348] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.864568 18365 raft_consensus.cc:374] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:55.864590 18365 raft_consensus.cc:729] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe751b7e124c42f7ac1df1fe0e0cacd2, State: Initialized, Role: FOLLOWER
I20260620 14:22:55.864744 18365 consensus_queue.cc:260] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.865002 18365 ts_tablet_manager.cc:1428] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:22:55.864885 18363 raft_consensus.cc:348] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.865041 18363 raft_consensus.cc:374] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:55.865092 18363 raft_consensus.cc:729] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1eba7717e048dcb0be3d73a0fd128e, State: Initialized, Role: FOLLOWER
I20260620 14:22:55.865123 18218 heartbeater.cc:503] Master 127.0.98.62:42765 was elected leader, sending a full tablet report...
I20260620 14:22:55.865207 18363 consensus_queue.cc:260] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.865451 18363 ts_tablet_manager.cc:1428] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 14:22:55.865484 18087 heartbeater.cc:503] Master 127.0.98.62:42765 was elected leader, sending a full tablet report...
I20260620 14:22:55.866041 18364 raft_consensus.cc:348] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.866187 18364 raft_consensus.cc:374] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:55.866241 18364 raft_consensus.cc:729] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45b15473819438f8acb12e36fc780c8, State: Initialized, Role: FOLLOWER
I20260620 14:22:55.866355 18364 consensus_queue.cc:260] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.866636 18364 ts_tablet_manager.cc:1428] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:55.866690 18349 heartbeater.cc:503] Master 127.0.98.62:42765 was elected leader, sending a full tablet report...
I20260620 14:22:55.887804 18370 raft_consensus.cc:482] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:55.887954 18370 raft_consensus.cc:504] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.888346 18370 leader_election.cc:288] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e45b15473819438f8acb12e36fc780c8 (127.0.98.3:37317), fe751b7e124c42f7ac1df1fe0e0cacd2 (127.0.98.2:41729)
I20260620 14:22:55.892045 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.888908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52701 (local address 127.0.98.2:41729)
0620 14:22:55.888972 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.888974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.888982 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:22:55.889018 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.889019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.889031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.889076 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.889233 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.890047 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.891121 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.891686 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.891750 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.891761 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.891955 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.891993 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:22:55.892083 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.888603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41729 (local address 127.0.98.1:52701)
0620 14:22:55.888762 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.888773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.888833 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.889119 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.889119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.889121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.889177 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.889178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.890127 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.890128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.891009 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.891011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.891793 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.891794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.891875 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.891879 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.891932 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.891970 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260620 14:22:55.892376 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.888497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.98.1:57185)
0620 14:22:55.888730 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.888743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:55.888801 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:55.889047 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:55.889048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:55.889050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:55.889124 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.889126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.890108 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.890109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.891343 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:55.891346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.892128 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.892129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:55.892211 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.892215 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:55.892277 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:22:55.892318 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260620 14:22:55.892405 18172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c4495928004c33802a4593867148ec" candidate_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" is_pre_election: true
I20260620 14:22:55.892563 18172 raft_consensus.cc:2454] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1eba7717e048dcb0be3d73a0fd128e in term 0.
I20260620 14:22:55.892814 17976 leader_election.cc:302] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1eba7717e048dcb0be3d73a0fd128e, fe751b7e124c42f7ac1df1fe0e0cacd2; no voters:
I20260620 14:22:55.892849 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.888671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57185 (local address 127.0.98.3:37317)
0620 14:22:55.888743 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.888745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.888790 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:22:55.888900 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.888901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.888910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.888985 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:55.889192 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.890002 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.891435 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.892043 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.892114 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.892124 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.892708 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.892769 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:22:55.892971 18370 raft_consensus.cc:2781] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:55.893019 18370 raft_consensus.cc:482] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:55.893061 18370 raft_consensus.cc:3037] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:55.893167 18303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c4495928004c33802a4593867148ec" candidate_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45b15473819438f8acb12e36fc780c8" is_pre_election: true
I20260620 14:22:55.893319 18303 raft_consensus.cc:2454] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1eba7717e048dcb0be3d73a0fd128e in term 0.
I20260620 14:22:55.894045 18370 raft_consensus.cc:504] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.894228 18370 leader_election.cc:288] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [CANDIDATE]: Term 1 election: Requested vote from peers e45b15473819438f8acb12e36fc780c8 (127.0.98.3:37317), fe751b7e124c42f7ac1df1fe0e0cacd2 (127.0.98.2:41729)
I20260620 14:22:55.894361 18303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c4495928004c33802a4593867148ec" candidate_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e45b15473819438f8acb12e36fc780c8"
I20260620 14:22:55.894381 18172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c4495928004c33802a4593867148ec" candidate_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2"
I20260620 14:22:55.894441 18303 raft_consensus.cc:3037] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:55.894444 18172 raft_consensus.cc:3037] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:55.895357 18172 raft_consensus.cc:2454] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1eba7717e048dcb0be3d73a0fd128e in term 1.
I20260620 14:22:55.895357 18303 raft_consensus.cc:2454] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1eba7717e048dcb0be3d73a0fd128e in term 1.
I20260620 14:22:55.895581 17976 leader_election.cc:302] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1eba7717e048dcb0be3d73a0fd128e, fe751b7e124c42f7ac1df1fe0e0cacd2; no voters:
I20260620 14:22:55.895692 18370 raft_consensus.cc:2781] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:55.895879 18370 raft_consensus.cc:686] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [term 1 LEADER]: Becoming Leader. State: Replica: 2e1eba7717e048dcb0be3d73a0fd128e, State: Running, Role: LEADER
I20260620 14:22:55.895975 18370 consensus_queue.cc:237] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } }
I20260620 14:22:55.896938 17898 catalog_manager.cc:5495] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e1eba7717e048dcb0be3d73a0fd128e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1eba7717e048dcb0be3d73a0fd128e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 } health_report { overall_health: UNKNOWN } } }
I20260620 14:22:55.902292 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.899916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.98.1:42793)
0620 14:22:55.899975 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:55.899977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:55.899985 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:22:55.900057 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:55.900058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:55.900075 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:55.900175 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:55.900272 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.900765 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.901505 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:55.901712 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:55.901796 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:55.901957 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:55.902070 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:22:55.902163 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:55.902183 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:22:55.902202 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:22:55.904071 18303 raft_consensus.cc:1261] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 1 FOLLOWER]: Refusing update from remote peer 2e1eba7717e048dcb0be3d73a0fd128e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:55.904073 18172 raft_consensus.cc:1261] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 1 FOLLOWER]: Refusing update from remote peer 2e1eba7717e048dcb0be3d73a0fd128e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:55.904458 18373 consensus_queue.cc:1035] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe751b7e124c42f7ac1df1fe0e0cacd2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:55.904590 18370 consensus_queue.cc:1035] T 14c4495928004c33802a4593867148ec P 2e1eba7717e048dcb0be3d73a0fd128e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e45b15473819438f8acb12e36fc780c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:55.907195 18376 mvcc.cc:205] Tried to move back new op lower bound from 7298930179700326400 to 7298930179670679552. Current Snapshot: MvccSnapshot[applied={T|T < 7298930179700326400}]
I20260620 14:22:55.908124 18378 mvcc.cc:205] Tried to move back new op lower bound from 7298930179700326400 to 7298930179670679552. Current Snapshot: MvccSnapshot[applied={T|T < 7298930179700326400 or (T in {7298930179700326400})}]
I20260620 14:22:55.909286 18377 mvcc.cc:205] Tried to move back new op lower bound from 7298930179700326400 to 7298930179670679552. Current Snapshot: MvccSnapshot[applied={T|T < 7298930179700326400 or (T in {7298930179700326400})}]
I20260620 14:22:55.909461 17898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:55.909539 17898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:55.910281 17898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41a4942aff2d4f1ab805c53fbb175b56: Sending DeleteTablet for 3 replicas of tablet 14c4495928004c33802a4593867148ec
I20260620 14:22:55.910636 18021 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c4495928004c33802a4593867148ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:55 UTC) from {username='slave'} at 127.0.0.1:56966
I20260620 14:22:55.910650 18283 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c4495928004c33802a4593867148ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:55 UTC) from {username='slave'} at 127.0.0.1:36186
I20260620 14:22:55.910874 18388 tablet_replica.cc:330] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: stopping tablet replica
I20260620 14:22:55.910863 18152 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c4495928004c33802a4593867148ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:55 UTC) from {username='slave'} at 127.0.0.1:50794
I20260620 14:22:55.911031 18388 raft_consensus.cc:2229] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:55.911073 18389 tablet_replica.cc:330] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: stopping tablet replica
I20260620 14:22:55.911154 18389 raft_consensus.cc:2229] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:55.911173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17959
I20260620 14:22:55.911835 18389 raft_consensus.cc:2258] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:55.911937 18388 raft_consensus.cc:2258] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:55.912312 18389 ts_tablet_manager.cc:1905] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:55.914168 18389 ts_tablet_manager.cc:1918] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:55.914251 18389 log.cc:1196] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/14c4495928004c33802a4593867148ec
I20260620 14:22:55.914530 18389 ts_tablet_manager.cc:1939] T 14c4495928004c33802a4593867148ec P fe751b7e124c42f7ac1df1fe0e0cacd2: Deleting consensus metadata
I20260620 14:22:55.914933 17885 catalog_manager.cc:4841] TS fe751b7e124c42f7ac1df1fe0e0cacd2 (127.0.98.2:41729): tablet 14c4495928004c33802a4593867148ec (table test-table [id=0e2c69da5a4d4a64ac574ce27b47ee68]) successfully deleted
I20260620 14:22:55.917727 18388 ts_tablet_manager.cc:1905] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:22:55.919183 17883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42793 (error 108)
I20260620 14:22:55.919621 18388 ts_tablet_manager.cc:1918] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:55.919695 18388 log.cc:1196] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/14c4495928004c33802a4593867148ec
I20260620 14:22:55.919968 18388 ts_tablet_manager.cc:1939] T 14c4495928004c33802a4593867148ec P e45b15473819438f8acb12e36fc780c8: Deleting consensus metadata
I20260620 14:22:55.920177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 18090
I20260620 14:22:55.921589 17885 catalog_manager.cc:4841] TS e45b15473819438f8acb12e36fc780c8 (127.0.98.3:37317): tablet 14c4495928004c33802a4593867148ec (table test-table [id=0e2c69da5a4d4a64ac574ce27b47ee68]) successfully deleted
I20260620 14:22:55.922442 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:55.922301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42793 (local address 127.0.0.1:56984)
0620 14:22:55.922366 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:55.922414 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:22:55.922570 17883 catalog_manager.cc:4584] TS 2e1eba7717e048dcb0be3d73a0fd128e (127.0.98.1:42793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14c4495928004c33802a4593867148ec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42793: connect: Connection refused (error 111)
I20260620 14:22:55.926208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 18221
I20260620 14:22:55.932478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 17868
2026-06-20T14:22:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (718 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:22:55 dist-test-slave-h6n5 krb5kdc[18393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:22:55 dist-test-slave-h6n5 krb5kdc[18393](info): set up 2 sockets
Jun 20 14:22:55 dist-test-slave-h6n5 krb5kdc[18393](info): commencing operation
krb5kdc: starting...
W20260620 14:22:56.987439 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:22:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:22:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:22:57.106148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:57.187781 18406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:57.188040 18406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:57.188081 18406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:57.189262 18406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:57.189340 18406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:57.189364 18406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:57.189388 18406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:57.189416 18406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:57.190862 18406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:57.191151 18406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:57.191391 18406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:57.196343 18412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:57.196280 18411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:57.196275 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:57.196550 18406 server_base.cc:1034] running on GCE node
I20260620 14:22:57.196798 18406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:57.197108 18406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:57.198298 18406 hybrid_clock.cc:648] HybridClock initialized: now 1781965377198262 us; error 50 us; skew 500 ppm
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:57.201057 18406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:22:57.201450 18406 webserver.cc:460] Webserver started at http://127.0.98.62:38929/ using document root <none> and password file <none>
I20260620 14:22:57.201628 18406 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:57.201676 18406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:57.201766 18406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:57.202783 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "040c04a2ad5647ffbbffe187057efc3a"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "46769cb55f7a77e0cf0f40a890d8a5c3"
server_key_iv: "220dbc87bf7d9894a61e40dbaa1e0961"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.203076 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "040c04a2ad5647ffbbffe187057efc3a"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "46769cb55f7a77e0cf0f40a890d8a5c3"
server_key_iv: "220dbc87bf7d9894a61e40dbaa1e0961"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.204493 18406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:22:57.205193 18421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.205382 18406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:22:57.205476 18406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "040c04a2ad5647ffbbffe187057efc3a"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "46769cb55f7a77e0cf0f40a890d8a5c3"
server_key_iv: "220dbc87bf7d9894a61e40dbaa1e0961"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.205554 18406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:57.227604 18406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:57.230032 18406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:57.230142 18406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:57.234066 18406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34337
I20260620 14:22:57.234122 18473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34337 every 8 connection(s)
I20260620 14:22:57.234473 18406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:57.235390 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:57.237713 18474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a: Bootstrap starting.
I20260620 14:22:57.238334 18474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:57.238641 18474 log.cc:824] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:57.239521 18474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a: No bootstrap required, opened a new log
I20260620 14:22:57.240916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 18406
I20260620 14:22:57.241070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:22:57.241257 392 external_mini_cluster.cc:1428] Setting key 6c5cb69f75505dcae5256a82baf28fe9
I20260620 14:22:57.241364 18474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040c04a2ad5647ffbbffe187057efc3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34337 } }
I20260620 14:22:57.241535 18474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:57.241588 18474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040c04a2ad5647ffbbffe187057efc3a, State: Initialized, Role: FOLLOWER
I20260620 14:22:57.241847 18474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040c04a2ad5647ffbbffe187057efc3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34337 } }
I20260620 14:22:57.241954 18474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:57.241988 18474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:57.242053 18474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:57.243218 18474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040c04a2ad5647ffbbffe187057efc3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34337 } }
I20260620 14:22:57.243356 18474 leader_election.cc:302] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 040c04a2ad5647ffbbffe187057efc3a; no voters:
I20260620 14:22:57.243531 18474 leader_election.cc:288] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:22:57.243597 18479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:57.243711 18479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [term 1 LEADER]: Becoming Leader. State: Replica: 040c04a2ad5647ffbbffe187057efc3a, State: Running, Role: LEADER
I20260620 14:22:57.243804 18479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040c04a2ad5647ffbbffe187057efc3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34337 } }
I20260620 14:22:57.243842 18474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:57.244120 18481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "040c04a2ad5647ffbbffe187057efc3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040c04a2ad5647ffbbffe187057efc3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34337 } } }
I20260620 14:22:57.244134 18480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040c04a2ad5647ffbbffe187057efc3a. Latest consensus state: current_term: 1 leader_uuid: "040c04a2ad5647ffbbffe187057efc3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040c04a2ad5647ffbbffe187057efc3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34337 } } }
I20260620 14:22:57.244313 18481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:57.244334 18480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:57.244664 18488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:57.245200 18488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:57.246706 18488 catalog_manager.cc:1347] Generated new cluster ID: b273e385acec4e72946c83b638e264a5
I20260620 14:22:57.246745 18488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:57.252492 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.241887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.0.98.62:34337)
0620 14:22:57.242098 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.242108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.242140 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:22:57.242256 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.242262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.242311 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.242421 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:57.243340 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.244749 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.245989 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.246576 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.247042 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.249148 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:57.249152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:57.249153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:57.249164 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:57.250853 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.251232 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.251235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.251236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.251349 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.251616 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.251617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.251618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.251854 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:57.252035 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.252101 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.252349 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260620 14:22:57.258190 18488 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:22:57.258834 18488 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:57.263170 18488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a: Generated new TSK 0
I20260620 14:22:57.263345 18488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:22:57.292438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34337
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:57.373530 18500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:57.373723 18500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:57.373746 18500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:57.374902 18500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:57.374944 18500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:57.374969 18500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:57.376475 18500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:57.376721 18500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:57.376965 18500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:57.381906 18506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:57.381896 18505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:57.381912 18500 server_base.cc:1034] running on GCE node
W20260620 14:22:57.381897 18508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:57.382375 18500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:57.382573 18500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:57.383705 18500 hybrid_clock.cc:648] HybridClock initialized: now 1781965377383695 us; error 49 us; skew 500 ppm
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:57.386240 18500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:22:57.386674 18500 webserver.cc:460] Webserver started at http://127.0.98.1:42883/ using document root <none> and password file <none>
I20260620 14:22:57.386901 18500 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:57.386945 18500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:57.387089 18500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:57.388188 18500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b37918a49a5421eb0ac6f69e2999e4e"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "3754b2d36f1548896e0f7ed1d98480c3"
server_key_iv: "ba89c5988e986ace443baf3bb53c23b6"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.388542 18500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b37918a49a5421eb0ac6f69e2999e4e"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "3754b2d36f1548896e0f7ed1d98480c3"
server_key_iv: "ba89c5988e986ace443baf3bb53c23b6"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.389818 18500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:22:57.390549 18515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.390727 18500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:22:57.390823 18500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "1b37918a49a5421eb0ac6f69e2999e4e"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "3754b2d36f1548896e0f7ed1d98480c3"
server_key_iv: "ba89c5988e986ace443baf3bb53c23b6"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.390901 18500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:57.423385 18500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:57.425904 18500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:57.426070 18500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:57.426326 18500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:57.426653 18500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:57.426708 18500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.426749 18500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:57.426779 18500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.432636 18500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43069
I20260620 14:22:57.432698 18628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43069 every 8 connection(s)
I20260620 14:22:57.432993 18500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:57.437685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 18500
I20260620 14:22:57.437808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:22:57.438069 392 external_mini_cluster.cc:1428] Setting key 1d7e98f9453f62a3442554fbf3aeaae9
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:57.443460 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.433684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51277 (local address 127.0.98.62:34337)
0620 14:22:57.433755 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.433757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.433972 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:22:57.434882 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.434883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.434892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.434952 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:57.435672 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.436194 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.437278 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.437479 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.437545 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.440302 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:57.440304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:57.440304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:57.440314 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:57.441858 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.442453 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.442454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.442454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.442516 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.442778 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.442780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.442780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.442901 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:57.442973 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.443284 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.443370 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:22:57.443711 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.433650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34337 (local address 127.0.98.1:51277)
0620 14:22:57.433836 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:57.433879 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:57.434796 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:57.435021 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:57.435025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:57.435307 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:57.435504 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.435508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.436337 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.436338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.437186 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.437188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.437589 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.437591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.437762 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.438462 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:57.438473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:57.440194 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:57.442067 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:57.442076 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:57.442079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:57.442353 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:57.442589 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:57.442590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:57.442590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:57.442707 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:57.442975 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:57.442982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:57.443138 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:22:57.443251 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:22:57.443416 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260620 14:22:57.444546 18629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34337
I20260620 14:22:57.444690 18629 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:57.444880 18629 heartbeater.cc:511] Master 127.0.98.62:34337 requested a full tablet report, sending...
I20260620 14:22:57.445468 18438 ts_manager.cc:194] Registered new tserver with Master: 1b37918a49a5421eb0ac6f69e2999e4e (127.0.98.1:43069)
I20260620 14:22:57.446066 18438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51277
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:22:57.462628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34337
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:57.543664 18634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:57.543931 18634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:57.543959 18634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:57.545250 18634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:57.545295 18634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:57.545323 18634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:57.546851 18634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:57.547039 18634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:57.547215 18634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:57.552268 18639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:57.552281 18642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:57.552285 18640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:57.553067 18634 server_base.cc:1034] running on GCE node
I20260620 14:22:57.553335 18634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:57.553578 18634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:57.554744 18634 hybrid_clock.cc:648] HybridClock initialized: now 1781965377554739 us; error 35 us; skew 500 ppm
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:57.557408 18634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:22:57.558018 18634 webserver.cc:460] Webserver started at http://127.0.98.2:36965/ using document root <none> and password file <none>
I20260620 14:22:57.558261 18634 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:57.558333 18634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:57.558486 18634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:57.559525 18634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a6c296525d046df84c6cbec7b41703f"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "648d0cd031aef52c8279ae49f5c8260a"
server_key_iv: "976b7931760af6295825e160730fcb77"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.559983 18634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a6c296525d046df84c6cbec7b41703f"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "648d0cd031aef52c8279ae49f5c8260a"
server_key_iv: "976b7931760af6295825e160730fcb77"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.561313 18634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:57.562094 18649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.562326 18634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:57.562426 18634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "1a6c296525d046df84c6cbec7b41703f"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "648d0cd031aef52c8279ae49f5c8260a"
server_key_iv: "976b7931760af6295825e160730fcb77"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.562479 18634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:57.578544 18634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:57.580715 18634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:57.580826 18634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:57.581007 18634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:57.581352 18634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:57.581388 18634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.581411 18634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:57.581460 18634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.587217 18634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45633
I20260620 14:22:57.587260 18762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45633 every 8 connection(s)
I20260620 14:22:57.587648 18634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:57.597402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 18634
I20260620 14:22:57.597536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:22:57.597751 392 external_mini_cluster.cc:1428] Setting key 4ea726fa1b84df06a8538463dfe20c20
I20260620 14:22:57.597860 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.588406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37039 (local address 127.0.98.62:34337)
0620 14:22:57.588453 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.588455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.588699 (+ 244us) server_negotiation.cc:407] Connection header received
0620 14:22:57.589934 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.589936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.589951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.590023 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:57.590780 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.591294 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.592330 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.592517 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.592573 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.594917 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:57.594919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:57.594919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:57.594925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:57.596417 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.596840 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.596842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.596842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.596901 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.597164 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.597165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.597165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.597279 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:57.597357 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.597680 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.597770 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:22:57.598126 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.588379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34337 (local address 127.0.98.2:37039)
0620 14:22:57.588575 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:57.588609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:57.589808 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:57.590124 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:57.590129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:57.590501 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:57.590704 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.590709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.591378 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.591379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.592256 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.592259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.592600 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.592601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.592712 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.593219 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:57.593228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:57.594726 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:57.596517 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:57.596524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:57.596526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:57.596753 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:57.596982 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:57.596983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:57.596983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:57.597094 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:57.597362 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:57.597368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:57.597547 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 14:22:57.597659 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:22:57.597819 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260620 14:22:57.598765 18763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34337
I20260620 14:22:57.598850 18763 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:57.599043 18763 heartbeater.cc:511] Master 127.0.98.62:34337 requested a full tablet report, sending...
I20260620 14:22:57.599437 18438 ts_manager.cc:194] Registered new tserver with Master: 1a6c296525d046df84c6cbec7b41703f (127.0.98.2:45633)
I20260620 14:22:57.599977 18438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37039
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:22:57.622407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34337
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:57.703137 18768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:57.703413 18768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:57.703460 18768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:57.705041 18768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:57.705127 18768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:57.705181 18768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:57.706792 18768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:57.707085 18768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:57.707335 18768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:57.712200 18773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:57.712276 18774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:57.712283 18776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:57.712847 18768 server_base.cc:1034] running on GCE node
I20260620 14:22:57.713110 18768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:57.713380 18768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:57.714542 18768 hybrid_clock.cc:648] HybridClock initialized: now 1781965377714540 us; error 40 us; skew 500 ppm
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:22:57.717121 18768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:22:57.717501 18768 webserver.cc:460] Webserver started at http://127.0.98.3:38271/ using document root <none> and password file <none>
I20260620 14:22:57.717706 18768 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:57.717748 18768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:57.717839 18768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:57.718830 18768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "7af26e3f60e44ada824bf004d0175841"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "a6ca9143b563a6caf17384976a5ed45f"
server_key_iv: "0cdb24924292be0ebd8f586939329baf"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.719121 18768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7af26e3f60e44ada824bf004d0175841"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "a6ca9143b563a6caf17384976a5ed45f"
server_key_iv: "0cdb24924292be0ebd8f586939329baf"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.720531 18768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:22:57.721294 18783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.721504 18768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:22:57.721577 18768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "7af26e3f60e44ada824bf004d0175841"
format_stamp: "Formatted at 2026-06-20 14:22:57 on dist-test-slave-h6n5"
server_key: "a6ca9143b563a6caf17384976a5ed45f"
server_key_iv: "0cdb24924292be0ebd8f586939329baf"
server_key_version: "encryptionkey@0"
I20260620 14:22:57.721671 18768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:57.728353 18768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:57.730225 18768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:57.730370 18768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:57.730594 18768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:57.730901 18768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:57.730954 18768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.730996 18768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:57.731024 18768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:57.736564 18768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32997
I20260620 14:22:57.736657 18896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32997 every 8 connection(s)
I20260620 14:22:57.736966 18768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:22:57 dist-test-slave-h6n5 krb5kdc[18393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:22:57.746805 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.737805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36763 (local address 127.0.98.62:34337)
0620 14:22:57.737875 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.737877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.738053 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:22:57.739010 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.739011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.739021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.739076 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:57.739808 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.740308 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.741373 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.741561 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.741621 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.744290 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:57.744292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:57.744293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:57.744302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:57.745640 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.746037 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.746038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.746039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.746089 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.746312 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.746312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.746313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.746412 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:57.746462 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.746682 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.746739 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:22:57.747025 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.737706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34337 (local address 127.0.98.3:36763)
0620 14:22:57.737939 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:57.737974 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:57.738903 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:57.739183 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:57.739186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:57.739452 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:57.739717 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.739723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.740406 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.740407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.741285 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.741288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.741639 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.741640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.741740 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.742520 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:22:57.742529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:57.744119 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:57.745728 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:57.745734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:57.745736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:57.745971 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:57.746147 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:22:57.746148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:22:57.746148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:22:57.746255 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:22:57.746466 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:57.746470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:57.746581 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 14:22:57.746661 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:22:57.746796 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
I20260620 14:22:57.747032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 18768
I20260620 14:22:57.747128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:22:57.747337 392 external_mini_cluster.cc:1428] Setting key 8ce0bb699f498ce0db59aebd4074fe75
I20260620 14:22:57.747860 18897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34337
I20260620 14:22:57.747972 18897 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:57.748169 18897 heartbeater.cc:511] Master 127.0.98.62:34337 requested a full tablet report, sending...
I20260620 14:22:57.748603 18437 ts_manager.cc:194] Registered new tserver with Master: 7af26e3f60e44ada824bf004d0175841 (127.0.98.3:32997)
I20260620 14:22:57.748971 18437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36763
I20260620 14:22:57.759253 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:57.768405 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.760640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.98.62:34337)
0620 14:22:57.760745 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.760748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.760761 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:22:57.760841 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.760842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.760850 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.760900 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:57.761461 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.761938 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.762643 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.762838 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.762892 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.764818 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:57.764820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:57.764820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:22:57.764828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:57.766622 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.767296 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.767297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.767297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.767368 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:22:57.767684 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:22:57.767685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:22:57.767685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:22:57.767885 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:57.768007 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.768193 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.768270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:22:57.771287 18437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:22:57.773912 18437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:22:57.781466 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.776600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43069 (local address 127.0.0.1:41102)
0620 14:22:57.776809 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:57.776854 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:57.776917 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:57.777406 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:57.777409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:57.777411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:57.777506 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.777508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.778973 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.778974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.780275 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.780277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.781186 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.781187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.781278 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.781293 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:22:57.781337 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:22:57.781383 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":46,"threads_started":1}
I20260620 14:22:57.781950 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.776648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32997 (local address 127.0.0.1:41814)
0620 14:22:57.776924 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:57.776935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:57.776991 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:57.777749 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:57.777750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:57.777752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:57.777796 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.777798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.779322 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.779323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.780619 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.780622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.781717 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.781718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.781788 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.781792 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:57.781842 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:22:57.781881 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":90,"threads_started":1}
I20260620 14:22:57.782275 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.776878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.98.3:32997)
0620 14:22:57.777603 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.777608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.777618 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:22:57.777656 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.777657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.777674 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.777750 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:57.777960 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.779213 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.780733 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.781550 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.781621 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.781680 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.782157 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.782200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":81,"threads_started":1}
I20260620 14:22:57.782974 18831 tablet_service.cc:1467] Processing CreateTablet for tablet 869701537478424b887b8e02b197b50e (DEFAULT_TABLE table=test-table [id=c545dbc11e6048fca3b67fc6c57843ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:57.783339 18831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869701537478424b887b8e02b197b50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:57.784186 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.777261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.0.98.2:45633)
0620 14:22:57.780441 (+ 3180us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.780446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.780454 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:22:57.780490 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.780492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.780506 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.780577 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:57.780789 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.781811 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.783079 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.783791 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.783869 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.783918 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.784069 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.784112 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3144,"thread_start_us":113,"threads_started":1}
I20260620 14:22:57.784184 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.776805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45633 (local address 127.0.0.1:50006)
0620 14:22:57.777845 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:57.777856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:57.777898 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:57.780659 (+ 2761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:57.780660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:57.780662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:57.780717 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.780719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.781913 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.781914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.782986 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:57.782989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.783885 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.783886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:57.783959 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.783964 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:22:57.784050 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:22:57.784090 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":76,"threads_started":1}
I20260620 14:22:57.786990 18914 tablet_bootstrap.cc:492] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Bootstrap starting.
I20260620 14:22:57.787142 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:57.776706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.98.1:43069)
0620 14:22:57.777256 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:57.777261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:57.777269 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:22:57.777301 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:57.777302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:57.777314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:57.777403 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:57.777600 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.778854 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.780359 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:57.781090 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:57.781849 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:57.781915 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:57.781940 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 14:22:57.787055 (+ 5115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":75,"threads_started":1}
I20260620 14:22:57.787892 18563 tablet_service.cc:1467] Processing CreateTablet for tablet 869701537478424b887b8e02b197b50e (DEFAULT_TABLE table=test-table [id=c545dbc11e6048fca3b67fc6c57843ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:57.788070 18914 tablet_bootstrap.cc:654] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:57.788228 18563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869701537478424b887b8e02b197b50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:57.788658 18914 log.cc:824] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:57.788964 18697 tablet_service.cc:1467] Processing CreateTablet for tablet 869701537478424b887b8e02b197b50e (DEFAULT_TABLE table=test-table [id=c545dbc11e6048fca3b67fc6c57843ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:57.789283 18697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869701537478424b887b8e02b197b50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:57.789408 18914 tablet_bootstrap.cc:492] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: No bootstrap required, opened a new log
I20260620 14:22:57.789477 18914 ts_tablet_manager.cc:1397] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:22:57.790990 18914 raft_consensus.cc:348] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:57.791157 18914 raft_consensus.cc:374] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:57.791201 18914 raft_consensus.cc:729] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af26e3f60e44ada824bf004d0175841, State: Initialized, Role: FOLLOWER
I20260620 14:22:57.791311 18914 consensus_queue.cc:260] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:57.791574 18916 tablet_bootstrap.cc:492] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e: Bootstrap starting.
I20260620 14:22:57.791628 18914 ts_tablet_manager.cc:1428] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:22:57.791872 18897 heartbeater.cc:503] Master 127.0.98.62:34337 was elected leader, sending a full tablet report...
I20260620 14:22:57.792236 18916 tablet_bootstrap.cc:654] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:57.792464 18918 tablet_bootstrap.cc:492] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Bootstrap starting.
I20260620 14:22:57.792847 18916 log.cc:824] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:57.793237 18918 tablet_bootstrap.cc:654] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:57.793809 18916 tablet_bootstrap.cc:492] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e: No bootstrap required, opened a new log
I20260620 14:22:57.793862 18918 log.cc:824] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:57.793889 18916 ts_tablet_manager.cc:1397] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:22:57.794780 18918 tablet_bootstrap.cc:492] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: No bootstrap required, opened a new log
I20260620 14:22:57.794847 18918 ts_tablet_manager.cc:1397] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:22:57.796137 18916 raft_consensus.cc:348] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:57.796344 18916 raft_consensus.cc:374] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:57.796388 18916 raft_consensus.cc:729] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b37918a49a5421eb0ac6f69e2999e4e, State: Initialized, Role: FOLLOWER
I20260620 14:22:57.796502 18916 consensus_queue.cc:260] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:57.796770 18916 ts_tablet_manager.cc:1428] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:57.797168 18629 heartbeater.cc:503] Master 127.0.98.62:34337 was elected leader, sending a full tablet report...
I20260620 14:22:57.797158 18918 raft_consensus.cc:348] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:57.797338 18918 raft_consensus.cc:374] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:57.797389 18918 raft_consensus.cc:729] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6c296525d046df84c6cbec7b41703f, State: Initialized, Role: FOLLOWER
I20260620 14:22:57.797502 18918 consensus_queue.cc:260] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:57.797791 18918 ts_tablet_manager.cc:1428] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:22:57.797816 18763 heartbeater.cc:503] Master 127.0.98.62:34337 was elected leader, sending a full tablet report...
W20260620 14:22:57.838248 18764 tablet.cc:2369] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:22:57.933677 18630 tablet.cc:2369] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:22:57.987594 18898 tablet.cc:2369] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:22:58.025334 18921 raft_consensus.cc:482] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:58.025507 18921 raft_consensus.cc:504] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:58.025887 18921 leader_election.cc:288] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a6c296525d046df84c6cbec7b41703f (127.0.98.2:45633), 7af26e3f60e44ada824bf004d0175841 (127.0.98.3:32997)
I20260620 14:22:58.029953 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.026213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43913 (local address 127.0.98.3:32997)
0620 14:22:58.026254 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.026256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.026307 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:22:58.026382 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.026383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.026392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.026443 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:58.026652 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.027423 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.028973 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.029571 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.029651 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.029661 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.029843 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.029884 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:22:58.029954 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.026170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38367 (local address 127.0.98.2:45633)
0620 14:22:58.026220 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.026222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.026308 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:22:58.026365 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.026366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.026379 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.026443 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:58.026649 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.027393 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.028973 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.029571 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.029650 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.029661 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.029842 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.029880 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:22:58.029990 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.026074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32997 (local address 127.0.98.1:43913)
0620 14:22:58.026231 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.026242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.026306 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.026469 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.026470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.026473 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:58.026569 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.026571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.027499 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.027500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.028864 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.028867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.029666 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.029667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.029756 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.029762 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.029823 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.029877 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":115,"thread_start_us":52,"threads_started":1}
I20260620 14:22:58.030001 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.026074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45633 (local address 127.0.98.1:38367)
0620 14:22:58.026230 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.026242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.026306 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.026469 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.026470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.026473 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:58.026575 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.026576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.027499 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.027500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.028864 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.028867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.029664 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.029666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.029756 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.029760 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.029823 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.029877 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":75,"thread_start_us":70,"threads_started":1}
I20260620 14:22:58.030337 18851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869701537478424b887b8e02b197b50e" candidate_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af26e3f60e44ada824bf004d0175841" is_pre_election: true
I20260620 14:22:58.030484 18851 raft_consensus.cc:2454] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b37918a49a5421eb0ac6f69e2999e4e in term 0.
I20260620 14:22:58.030601 18717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869701537478424b887b8e02b197b50e" candidate_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6c296525d046df84c6cbec7b41703f" is_pre_election: true
I20260620 14:22:58.030772 18717 raft_consensus.cc:2454] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b37918a49a5421eb0ac6f69e2999e4e in term 0.
I20260620 14:22:58.030774 18518 leader_election.cc:302] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b37918a49a5421eb0ac6f69e2999e4e, 7af26e3f60e44ada824bf004d0175841; no voters:
I20260620 14:22:58.030910 18921 raft_consensus.cc:2781] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:58.030990 18921 raft_consensus.cc:482] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:58.031015 18921 raft_consensus.cc:3037] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:58.031744 18921 raft_consensus.cc:504] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:58.031888 18921 leader_election.cc:288] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [CANDIDATE]: Term 1 election: Requested vote from peers 1a6c296525d046df84c6cbec7b41703f (127.0.98.2:45633), 7af26e3f60e44ada824bf004d0175841 (127.0.98.3:32997)
I20260620 14:22:58.032028 18851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869701537478424b887b8e02b197b50e" candidate_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af26e3f60e44ada824bf004d0175841"
I20260620 14:22:58.032114 18717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869701537478424b887b8e02b197b50e" candidate_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a6c296525d046df84c6cbec7b41703f"
I20260620 14:22:58.032142 18851 raft_consensus.cc:3037] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:58.032181 18717 raft_consensus.cc:3037] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:58.033038 18851 raft_consensus.cc:2454] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b37918a49a5421eb0ac6f69e2999e4e in term 1.
I20260620 14:22:58.033038 18717 raft_consensus.cc:2454] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b37918a49a5421eb0ac6f69e2999e4e in term 1.
I20260620 14:22:58.033214 18518 leader_election.cc:302] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b37918a49a5421eb0ac6f69e2999e4e, 7af26e3f60e44ada824bf004d0175841; no voters:
I20260620 14:22:58.033310 18921 raft_consensus.cc:2781] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:58.033474 18921 raft_consensus.cc:686] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [term 1 LEADER]: Becoming Leader. State: Replica: 1b37918a49a5421eb0ac6f69e2999e4e, State: Running, Role: LEADER
I20260620 14:22:58.033568 18921 consensus_queue.cc:237] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } }
I20260620 14:22:58.034471 18437 catalog_manager.cc:5495] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b37918a49a5421eb0ac6f69e2999e4e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b37918a49a5421eb0ac6f69e2999e4e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } health_report { overall_health: UNKNOWN } } }
I20260620 14:22:58.069876 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.067746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.0.98.1:43069)
0620 14:22:58.067794 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.067796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.067805 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:22:58.067861 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.067862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.067881 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.067938 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:58.068053 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.068511 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.069211 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.069408 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.069461 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.069607 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:58.069690 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:22:58.069759 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.069808 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.069817 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:22:58.071712 18717 raft_consensus.cc:1261] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 1 FOLLOWER]: Refusing update from remote peer 1b37918a49a5421eb0ac6f69e2999e4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:58.071712 18851 raft_consensus.cc:1261] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 1 FOLLOWER]: Refusing update from remote peer 1b37918a49a5421eb0ac6f69e2999e4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:58.072074 18925 consensus_queue.cc:1035] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a6c296525d046df84c6cbec7b41703f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:58.072218 18921 consensus_queue.cc:1035] T 869701537478424b887b8e02b197b50e P 1b37918a49a5421eb0ac6f69e2999e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af26e3f60e44ada824bf004d0175841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:58.074487 18928 mvcc.cc:205] Tried to move back new op lower bound from 7298930188578783232 to 7298930188426313728. Current Snapshot: MvccSnapshot[applied={T|T < 7298930188578783232}]
I20260620 14:22:58.074848 18929 mvcc.cc:205] Tried to move back new op lower bound from 7298930188578783232 to 7298930188426313728. Current Snapshot: MvccSnapshot[applied={T|T < 7298930188578783232}]
I20260620 14:22:58.075074 18930 mvcc.cc:205] Tried to move back new op lower bound from 7298930188578783232 to 7298930188426313728. Current Snapshot: MvccSnapshot[applied={T|T < 7298930188578783232}]
I20260620 14:22:58.077147 18437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:58.077229 18437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:58.078027 18437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 040c04a2ad5647ffbbffe187057efc3a: Sending DeleteTablet for 3 replicas of tablet 869701537478424b887b8e02b197b50e
I20260620 14:22:58.078342 18563 tablet_service.cc:1514] Processing DeleteTablet for tablet 869701537478424b887b8e02b197b50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41102
I20260620 14:22:58.078389 18831 tablet_service.cc:1514] Processing DeleteTablet for tablet 869701537478424b887b8e02b197b50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41814
I20260620 14:22:58.078611 18697 tablet_service.cc:1514] Processing DeleteTablet for tablet 869701537478424b887b8e02b197b50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50006
I20260620 14:22:58.078624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 18500
I20260620 14:22:58.079222 18939 tablet_replica.cc:330] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: stopping tablet replica
I20260620 14:22:58.079317 18940 tablet_replica.cc:330] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: stopping tablet replica
I20260620 14:22:58.079329 18939 raft_consensus.cc:2229] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:58.079424 18940 raft_consensus.cc:2229] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:58.079439 18939 raft_consensus.cc:2258] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:58.079527 18940 raft_consensus.cc:2258] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:58.079918 18939 ts_tablet_manager.cc:1905] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:58.079933 18940 ts_tablet_manager.cc:1905] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:58.081408 18939 ts_tablet_manager.cc:1918] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:58.081506 18939 log.cc:1196] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/869701537478424b887b8e02b197b50e
I20260620 14:22:58.081777 18939 ts_tablet_manager.cc:1939] T 869701537478424b887b8e02b197b50e P 7af26e3f60e44ada824bf004d0175841: Deleting consensus metadata
I20260620 14:22:58.082226 18424 catalog_manager.cc:4841] TS 7af26e3f60e44ada824bf004d0175841 (127.0.98.3:32997): tablet 869701537478424b887b8e02b197b50e (table test-table [id=c545dbc11e6048fca3b67fc6c57843ad]) successfully deleted
I20260620 14:22:58.082376 18940 ts_tablet_manager.cc:1918] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:58.082448 18940 log.cc:1196] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/869701537478424b887b8e02b197b50e
I20260620 14:22:58.082701 18940 ts_tablet_manager.cc:1939] T 869701537478424b887b8e02b197b50e P 1a6c296525d046df84c6cbec7b41703f: Deleting consensus metadata
I20260620 14:22:58.083096 18422 catalog_manager.cc:4841] TS 1a6c296525d046df84c6cbec7b41703f (127.0.98.2:45633): tablet 869701537478424b887b8e02b197b50e (table test-table [id=c545dbc11e6048fca3b67fc6c57843ad]) successfully deleted
W20260620 14:22:58.085851 18422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43069 (error 108)
I20260620 14:22:58.086098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 18634
I20260620 14:22:58.086803 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.086267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43069 (local address 127.0.0.1:41116)
0620 14:22:58.086735 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.086777 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
W20260620 14:22:58.086952 18422 catalog_manager.cc:4584] TS 1b37918a49a5421eb0ac6f69e2999e4e (127.0.98.1:43069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 869701537478424b887b8e02b197b50e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43069: connect: Connection refused (error 111)
I20260620 14:22:58.090713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 18768
I20260620 14:22:58.095072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 18406
2026-06-20T14:22:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2162 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:22:58Z Disabled control of system clock
I20260620 14:22:58.122056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44745
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:22:58.201721 18945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:58.201998 18945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:58.202019 18945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:58.203125 18945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:22:58.203159 18945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:58.203172 18945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:58.203184 18945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:22:58.203226 18945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:22:58.204489 18945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:58.204666 18945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:58.204878 18945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:58.209647 18953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.209736 18945 server_base.cc:1034] running on GCE node
W20260620 14:22:58.209668 18951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.209668 18950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.210158 18945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:58.210464 18945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:58.211618 18945 hybrid_clock.cc:648] HybridClock initialized: now 1781965378211607 us; error 59 us; skew 500 ppm
I20260620 14:22:58.216042 18945 webserver.cc:460] Webserver started at http://127.0.98.62:33869/ using document root <none> and password file <none>
I20260620 14:22:58.216302 18945 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:58.216351 18945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:58.216501 18945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:58.217568 18945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "6a1173ad3aac4552ba9bfa90caa74fc7"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "04dba0438739871fd067d38bb547eb5d"
server_key_iv: "3d11ad6ea6c269521e61ff2191b4ad0f"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.217958 18945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a1173ad3aac4552ba9bfa90caa74fc7"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "04dba0438739871fd067d38bb547eb5d"
server_key_iv: "3d11ad6ea6c269521e61ff2191b4ad0f"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.219229 18945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:22:58.220012 18959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.220193 18945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.220264 18945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "6a1173ad3aac4552ba9bfa90caa74fc7"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "04dba0438739871fd067d38bb547eb5d"
server_key_iv: "3d11ad6ea6c269521e61ff2191b4ad0f"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.220346 18945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:58.231383 18945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:58.231693 18945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:58.231844 18945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:58.235632 18945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44745
I20260620 14:22:58.235653 19011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44745 every 8 connection(s)
I20260620 14:22:58.236016 18945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:22:58.236943 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:58.239197 19012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7: Bootstrap starting.
I20260620 14:22:58.239817 19012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:58.240059 19012 log.cc:824] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:58.240700 19012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7: No bootstrap required, opened a new log
I20260620 14:22:58.242274 19012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44745 } }
I20260620 14:22:58.242408 19012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:58.242430 19012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1173ad3aac4552ba9bfa90caa74fc7, State: Initialized, Role: FOLLOWER
I20260620 14:22:58.242569 19012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44745 } }
I20260620 14:22:58.242653 19012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:22:58.242676 19012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:22:58.242704 19012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:58.243155 19012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44745 } }
I20260620 14:22:58.243243 19012 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1173ad3aac4552ba9bfa90caa74fc7; no voters:
I20260620 14:22:58.243362 19012 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:22:58.243444 19015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:58.243638 19015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1173ad3aac4552ba9bfa90caa74fc7, State: Running, Role: LEADER
I20260620 14:22:58.243649 19012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:22:58.243763 19015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44745 } }
I20260620 14:22:58.244258 19016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44745 } } }
I20260620 14:22:58.244361 19016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:58.244493 19017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1173ad3aac4552ba9bfa90caa74fc7. Latest consensus state: current_term: 1 leader_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1173ad3aac4552ba9bfa90caa74fc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44745 } } }
I20260620 14:22:58.244575 19017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:22:58.244776 19024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:22:58.245260 19024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:22:58.246073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 18945
I20260620 14:22:58.246150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:22:58.246519 392 external_mini_cluster.cc:1428] Setting key 2ef18a69ad13ad35fa4df9a19f6dc177
I20260620 14:22:58.247269 19024 catalog_manager.cc:1347] Generated new cluster ID: 0475f75e1cce49eabdfc9a48c5710790
I20260620 14:22:58.247352 19024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:22:58.252646 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.247011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.98.62:44745)
0620 14:22:58.247235 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.247243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.247278 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:22:58.247397 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.247404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.247470 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.247678 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.247897 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.249280 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.250318 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.250941 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.251070 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.251762 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.251766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.251767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:58.251972 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.252012 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:58.252026 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.252099 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.252176 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.252255 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260620 14:22:58.259308 19024 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:22:58.259858 19024 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:22:58.262750 19024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7: Generated new TSK 0
I20260620 14:22:58.262916 19024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:22:58.265769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44745
--builtin_ntp_servers=127.0.98.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:22:58.345732 19036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:58.345928 19036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:58.345949 19036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:58.347062 19036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:58.347100 19036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:58.347126 19036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:22:58.348457 19036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:58.348687 19036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:58.348980 19036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:58.353889 19041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.353853 19044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.353852 19042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.354112 19036 server_base.cc:1034] running on GCE node
I20260620 14:22:58.354305 19036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:58.354559 19036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:58.355715 19036 hybrid_clock.cc:648] HybridClock initialized: now 1781965378355694 us; error 35 us; skew 500 ppm
I20260620 14:22:58.356992 19036 webserver.cc:460] Webserver started at http://127.0.98.1:45205/ using document root <none> and password file <none>
I20260620 14:22:58.357239 19036 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:58.357285 19036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:58.357431 19036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:58.358480 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "c210472b61864d6ea1b62fbe69de1bd6"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "1f72a6c0f8bd8cb748c01bb74740d1f2"
server_key_iv: "7a433b97f43a46c1e09b3e1b5cad621a"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.358840 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c210472b61864d6ea1b62fbe69de1bd6"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "1f72a6c0f8bd8cb748c01bb74740d1f2"
server_key_iv: "7a433b97f43a46c1e09b3e1b5cad621a"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.360337 19036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:22:58.361117 19050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.361310 19036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:58.361406 19036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "c210472b61864d6ea1b62fbe69de1bd6"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "1f72a6c0f8bd8cb748c01bb74740d1f2"
server_key_iv: "7a433b97f43a46c1e09b3e1b5cad621a"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.361475 19036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:58.392601 19036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:58.392874 19036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:58.392978 19036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:58.393172 19036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:58.393509 19036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:58.393543 19036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.393608 19036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:58.393636 19036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.399667 19036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40937
I20260620 14:22:58.399749 19163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40937 every 8 connection(s)
I20260620 14:22:58.400079 19036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:22:58.400781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 19036
I20260620 14:22:58.400895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:22:58.401115 392 external_mini_cluster.cc:1428] Setting key 35588cead297a69d62ea319d6d6afbd8
I20260620 14:22:58.402137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44745
--builtin_ntp_servers=127.0.98.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:22:58.405886 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.400817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53077 (local address 127.0.98.62:44745)
0620 14:22:58.400903 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.400906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.401226 (+ 320us) server_negotiation.cc:407] Connection header received
0620 14:22:58.401913 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.401914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.401925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.401966 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.402896 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.403398 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.404507 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.404722 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.404798 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.405587 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.405589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.405589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:58.405594 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.405610 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:58.405614 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.405667 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.405776 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.405807 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:22:58.406229 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.400775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44745 (local address 127.0.98.1:53077)
0620 14:22:58.401045 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.401093 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.401816 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.402041 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.402062 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.402072 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:58.402781 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.402788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.403528 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.403529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.404393 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.404395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.404817 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.404818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.404960 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.405450 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:22:58.405450 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:58.405478 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:22:58.405499 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:58.405674 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:58.405678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:58.405681 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.405779 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.405880 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":111,"threads_started":1}
I20260620 14:22:58.406677 19164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44745
I20260620 14:22:58.406770 19164 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:58.406963 19164 heartbeater.cc:511] Master 127.0.98.62:44745 requested a full tablet report, sending...
I20260620 14:22:58.407459 18976 ts_manager.cc:194] Registered new tserver with Master: c210472b61864d6ea1b62fbe69de1bd6 (127.0.98.1:40937)
I20260620 14:22:58.408107 18976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53077
W20260620 14:22:58.484050 19167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:58.484287 19167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:58.484325 19167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:58.485486 19167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:58.485548 19167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:58.485592 19167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:22:58.487021 19167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:58.487313 19167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:58.487663 19167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:58.492587 19172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.492643 19167 server_base.cc:1034] running on GCE node
W20260620 14:22:58.492605 19175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.492595 19173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.492955 19167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:58.493153 19167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:58.494314 19167 hybrid_clock.cc:648] HybridClock initialized: now 1781965378494300 us; error 27 us; skew 500 ppm
I20260620 14:22:58.495640 19167 webserver.cc:460] Webserver started at http://127.0.98.2:38897/ using document root <none> and password file <none>
I20260620 14:22:58.495857 19167 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:58.495899 19167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:58.495990 19167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:58.497015 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a228d0813a546d4ba51928db89b89f0"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "6bca424e8423ed7b200f54d7623717c1"
server_key_iv: "82aaf6777c33d2391b3b21ce9bb79778"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.497320 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a228d0813a546d4ba51928db89b89f0"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "6bca424e8423ed7b200f54d7623717c1"
server_key_iv: "82aaf6777c33d2391b3b21ce9bb79778"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.498593 19167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:22:58.499286 19181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.499439 19167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:22:58.499514 19167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "8a228d0813a546d4ba51928db89b89f0"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "6bca424e8423ed7b200f54d7623717c1"
server_key_iv: "82aaf6777c33d2391b3b21ce9bb79778"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.499616 19167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:58.521334 19167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:58.521612 19167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:58.521723 19167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:58.521934 19167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:58.522292 19167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:58.522326 19167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.522384 19167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:58.522414 19167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.528090 19167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41417
I20260620 14:22:58.528225 19294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41417 every 8 connection(s)
I20260620 14:22:58.528493 19167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:22:58.533798 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.529328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33647 (local address 127.0.98.62:44745)
0620 14:22:58.529404 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.529407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.529595 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:22:58.530142 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.530143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.530155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.530238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.530913 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.531422 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.532437 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.532641 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.532721 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.533503 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.533504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.533505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:58.533510 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.533524 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:58.533529 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.533597 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.533689 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.533717 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:22:58.534021 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.529259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44745 (local address 127.0.98.2:33647)
0620 14:22:58.529460 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.529505 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.530025 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.530288 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.530292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.530302 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:58.530808 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.530812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.531517 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.531518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.532341 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.532343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.532734 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.532735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.532869 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.533358 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:22:58.533359 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:58.533387 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:22:58.533408 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:58.533598 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:58.533604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:58.533607 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.533691 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.533807 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260620 14:22:58.534483 19295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44745
I20260620 14:22:58.534557 19295 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:58.534754 19295 heartbeater.cc:511] Master 127.0.98.62:44745 requested a full tablet report, sending...
I20260620 14:22:58.535142 18976 ts_manager.cc:194] Registered new tserver with Master: 8a228d0813a546d4ba51928db89b89f0 (127.0.98.2:41417)
I20260620 14:22:58.535516 18976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33647
I20260620 14:22:58.537057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 19167
I20260620 14:22:58.537139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:22:58.537353 392 external_mini_cluster.cc:1428] Setting key 41e06864ae09c7510a257efd481d3deb
I20260620 14:22:58.538182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44745
--builtin_ntp_servers=127.0.98.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:22:58.620172 19298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:22:58.620397 19298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:22:58.620435 19298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:22:58.621615 19298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:22:58.621680 19298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:22:58.621722 19298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:22:58.623127 19298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:22:58.623394 19298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:22:58.623757 19298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:22:58.628659 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.628659 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:22:58.628747 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:22:58.629011 19298 server_base.cc:1034] running on GCE node
I20260620 14:22:58.629233 19298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:22:58.629437 19298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:22:58.630630 19298 hybrid_clock.cc:648] HybridClock initialized: now 1781965378630610 us; error 27 us; skew 500 ppm
I20260620 14:22:58.631938 19298 webserver.cc:460] Webserver started at http://127.0.98.3:43293/ using document root <none> and password file <none>
I20260620 14:22:58.632265 19298 fs_manager.cc:359] Metadata directory not provided
I20260620 14:22:58.632339 19298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:22:58.632495 19298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:22:58.633663 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e850dac37f842b099985cda9f956954"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "c3d6f85f079d7c83ed77ef51f4614126"
server_key_iv: "29242251fd7b66bc5fbcfeddc87f62dc"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.634029 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e850dac37f842b099985cda9f956954"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "c3d6f85f079d7c83ed77ef51f4614126"
server_key_iv: "29242251fd7b66bc5fbcfeddc87f62dc"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.635388 19298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:22:58.636287 19312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.636518 19298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:22:58.636615 19298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "8e850dac37f842b099985cda9f956954"
format_stamp: "Formatted at 2026-06-20 14:22:58 on dist-test-slave-h6n5"
server_key: "c3d6f85f079d7c83ed77ef51f4614126"
server_key_iv: "29242251fd7b66bc5fbcfeddc87f62dc"
server_key_version: "encryptionkey@0"
I20260620 14:22:58.636706 19298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:22:58.655375 19298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:22:58.655761 19298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:22:58.655922 19298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:22:58.656173 19298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:22:58.656504 19298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:22:58.656556 19298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.656597 19298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:22:58.656625 19298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:22:58.662333 19298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43295
I20260620 14:22:58.662400 19425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43295 every 8 connection(s)
I20260620 14:22:58.662767 19298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:22:58.668191 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.663587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52925 (local address 127.0.98.62:44745)
0620 14:22:58.663661 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.663663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.663933 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:22:58.664409 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.664410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.664420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.664488 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.665263 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.665775 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.666861 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.667056 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.667148 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.667944 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.667945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.667945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:58.667950 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.667966 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:58.667969 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.668020 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.668102 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.668128 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:22:58.668375 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.663524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44745 (local address 127.0.98.3:52925)
0620 14:22:58.663783 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.663824 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.664302 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.664562 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.664566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.664574 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:22:58.665176 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.665182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.665856 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.665857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.666697 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.666699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.667159 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.667160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.667268 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.667806 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:22:58.667807 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:22:58.667835 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:22:58.667857 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:22:58.668023 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:22:58.668025 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:22:58.668027 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.668075 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.668152 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":158,"threads_started":1}
I20260620 14:22:58.668920 19426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44745
I20260620 14:22:58.669019 19426 heartbeater.cc:464] Registering TS with master...
I20260620 14:22:58.669168 19426 heartbeater.cc:511] Master 127.0.98.62:44745 requested a full tablet report, sending...
I20260620 14:22:58.669595 18976 ts_manager.cc:194] Registered new tserver with Master: 8e850dac37f842b099985cda9f956954 (127.0.98.3:43295)
I20260620 14:22:58.669934 18976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52925
I20260620 14:22:58.672847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 19298
I20260620 14:22:58.672961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:22:58.673168 392 external_mini_cluster.cc:1428] Setting key e9fcd2752db756a9c75dc57bde4b6b0c
I20260620 14:22:58.674090 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:22:58.678401 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.675540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.0.98.62:44745)
0620 14:22:58.675652 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.675654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.675762 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:22:58.675807 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.675808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.675819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.675867 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:22:58.676010 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.676474 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.677202 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.677383 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.677438 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.678066 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:22:58.678067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:22:58.678068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:22:58.678073 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:22:58.678089 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:22:58.678094 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:22:58.678171 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.678277 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.678287 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:22:58.680450 18976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:22:58.682098 18976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:22:58.688715 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.684521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41417 (local address 127.0.0.1:58408)
0620 14:22:58.684657 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.684699 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.684764 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.685006 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.685009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.685012 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:58.685086 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.685088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.686526 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.686527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.687663 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.687665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.688385 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.688386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.688460 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.688474 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.688560 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.688617 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":29,"thread_start_us":82,"threads_started":1}
I20260620 14:22:58.688735 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.684612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.0.98.2:41417)
0620 14:22:58.684829 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.684835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.684845 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:22:58.684884 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.684886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.684904 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.684972 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:58.685196 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.686434 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.687754 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.688304 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.688378 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.688471 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.688599 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.688638 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260620 14:22:58.688835 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.684527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40937 (local address 127.0.0.1:36316)
0620 14:22:58.684723 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.684733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.684788 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.685433 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.685434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.685436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:58.685507 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.685508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.686484 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.686485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.687678 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.687681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.688604 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.688605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.688654 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.688657 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.688686 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.688728 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":55,"thread_start_us":103,"threads_started":1}
I20260620 14:22:58.689270 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.684767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43295 (local address 127.0.0.1:47154)
0620 14:22:58.685068 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.685080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.685125 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.685825 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.685826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.685828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:58.685867 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.685868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.687096 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.687097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.688153 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.688155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.689067 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.689069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.689131 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.689135 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.689165 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.689210 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":69,"threads_started":1}
I20260620 14:22:58.689602 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.684917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.0.98.1:40937)
0620 14:22:58.685121 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.685125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.685135 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:22:58.685173 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.685175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.685190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.685326 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:58.685568 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.686397 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.687783 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.688529 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.689351 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.689457 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.689481 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.689521 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260620 14:22:58.689692 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.685195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.0.98.3:43295)
0620 14:22:58.685647 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.685651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.685661 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:22:58.685696 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.685698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.685733 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.685813 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:58.685954 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.687009 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.688255 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.688971 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.689451 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.689547 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.689571 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.689614 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":108,"threads_started":1}
I20260620 14:22:58.689828 19229 tablet_service.cc:1467] Processing CreateTablet for tablet 712e7ccdaa8d4760badfc9c0258b17d1 (DEFAULT_TABLE table=test-table [id=b25046cd3d8e407397ade5945e0136a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:58.690225 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712e7ccdaa8d4760badfc9c0258b17d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:58.690212 19098 tablet_service.cc:1467] Processing CreateTablet for tablet 712e7ccdaa8d4760badfc9c0258b17d1 (DEFAULT_TABLE table=test-table [id=b25046cd3d8e407397ade5945e0136a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:58.690294 19360 tablet_service.cc:1467] Processing CreateTablet for tablet 712e7ccdaa8d4760badfc9c0258b17d1 (DEFAULT_TABLE table=test-table [id=b25046cd3d8e407397ade5945e0136a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:22:58.690544 19098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712e7ccdaa8d4760badfc9c0258b17d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:58.690605 19360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712e7ccdaa8d4760badfc9c0258b17d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:22:58.693141 19440 tablet_bootstrap.cc:492] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6: Bootstrap starting.
I20260620 14:22:58.693917 19440 tablet_bootstrap.cc:654] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:58.693929 19442 tablet_bootstrap.cc:492] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Bootstrap starting.
I20260620 14:22:58.694278 19440 log.cc:824] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:58.694567 19442 tablet_bootstrap.cc:654] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:58.694898 19442 log.cc:824] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:58.695103 19440 tablet_bootstrap.cc:492] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6: No bootstrap required, opened a new log
I20260620 14:22:58.695176 19440 ts_tablet_manager.cc:1397] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:22:58.695192 19441 tablet_bootstrap.cc:492] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Bootstrap starting.
I20260620 14:22:58.695873 19441 tablet_bootstrap.cc:654] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Neither blocks nor log segments found. Creating new log.
I20260620 14:22:58.696210 19441 log.cc:824] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Log is configured to *not* fsync() on all Append() calls
I20260620 14:22:58.696774 19442 tablet_bootstrap.cc:492] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: No bootstrap required, opened a new log
I20260620 14:22:58.696848 19442 ts_tablet_manager.cc:1397] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:22:58.697989 19440 raft_consensus.cc:348] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.698184 19440 raft_consensus.cc:374] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:58.698233 19440 raft_consensus.cc:729] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c210472b61864d6ea1b62fbe69de1bd6, State: Initialized, Role: FOLLOWER
I20260620 14:22:58.698359 19440 consensus_queue.cc:260] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.698643 19440 ts_tablet_manager.cc:1428] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 14:22:58.698669 19164 heartbeater.cc:503] Master 127.0.98.62:44745 was elected leader, sending a full tablet report...
I20260620 14:22:58.699191 19441 tablet_bootstrap.cc:492] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: No bootstrap required, opened a new log
I20260620 14:22:58.699249 19441 ts_tablet_manager.cc:1397] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 14:22:58.700505 19442 raft_consensus.cc:348] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.700682 19442 raft_consensus.cc:374] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:58.700719 19442 raft_consensus.cc:729] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a228d0813a546d4ba51928db89b89f0, State: Initialized, Role: FOLLOWER
I20260620 14:22:58.700845 19442 consensus_queue.cc:260] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.701103 19442 ts_tablet_manager.cc:1428] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:22:58.701155 19295 heartbeater.cc:503] Master 127.0.98.62:44745 was elected leader, sending a full tablet report...
I20260620 14:22:58.701951 19441 raft_consensus.cc:348] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.702122 19441 raft_consensus.cc:374] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:22:58.702174 19441 raft_consensus.cc:729] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e850dac37f842b099985cda9f956954, State: Initialized, Role: FOLLOWER
I20260620 14:22:58.702288 19441 consensus_queue.cc:260] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.702567 19441 ts_tablet_manager.cc:1428] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 14:22:58.702622 19426 heartbeater.cc:503] Master 127.0.98.62:44745 was elected leader, sending a full tablet report...
I20260620 14:22:58.722383 19447 raft_consensus.cc:482] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:22:58.722564 19447 raft_consensus.cc:504] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.722944 19447 leader_election.cc:288] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c210472b61864d6ea1b62fbe69de1bd6 (127.0.98.1:40937), 8e850dac37f842b099985cda9f956954 (127.0.98.3:43295)
I20260620 14:22:58.725919 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.723095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40937 (local address 127.0.98.2:34899)
0620 14:22:58.723176 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.723189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.723235 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.723431 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.723432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.723434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:58.723511 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.723512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.724269 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.724270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.725107 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.725109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.725658 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.725659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.725735 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.725739 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.725809 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.725845 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:22:58.725926 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.723183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34899 (local address 127.0.98.1:40937)
0620 14:22:58.723231 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.723233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.723244 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:58.723330 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.723331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.723345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.723400 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:58.723635 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.724168 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.725213 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.725573 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.725658 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.725668 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.725827 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.725868 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:22:58.726101 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.723373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60985 (local address 127.0.98.3:43295)
0620 14:22:58.723434 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.723436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.723447 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:22:58.723541 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.723542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.723579 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.723627 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:22:58.723786 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.724457 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.725378 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.725748 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.725828 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.725838 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.725999 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.726037 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:22:58.726114 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.723215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43295 (local address 127.0.98.2:60985)
0620 14:22:58.723399 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.723410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:22:58.723459 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:22:58.723636 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:22:58.723637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:22:58.723638 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:22:58.723714 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.723715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.724523 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.724524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.725283 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:22:58.725285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.725830 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.725831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:22:58.725913 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.725916 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:22:58.725991 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:22:58.726029 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":9,"thread_start_us":119,"threads_started":1}
I20260620 14:22:58.726305 19118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712e7ccdaa8d4760badfc9c0258b17d1" candidate_uuid: "8a228d0813a546d4ba51928db89b89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c210472b61864d6ea1b62fbe69de1bd6" is_pre_election: true
I20260620 14:22:58.726478 19118 raft_consensus.cc:2454] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a228d0813a546d4ba51928db89b89f0 in term 0.
I20260620 14:22:58.726724 19183 leader_election.cc:302] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a228d0813a546d4ba51928db89b89f0, c210472b61864d6ea1b62fbe69de1bd6; no voters:
I20260620 14:22:58.726787 19380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712e7ccdaa8d4760badfc9c0258b17d1" candidate_uuid: "8a228d0813a546d4ba51928db89b89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e850dac37f842b099985cda9f956954" is_pre_election: true
I20260620 14:22:58.726867 19447 raft_consensus.cc:2781] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:22:58.726924 19447 raft_consensus.cc:482] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:22:58.726943 19380 raft_consensus.cc:2454] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a228d0813a546d4ba51928db89b89f0 in term 0.
I20260620 14:22:58.726958 19447 raft_consensus.cc:3037] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:58.727677 19447 raft_consensus.cc:504] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.727841 19447 leader_election.cc:288] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [CANDIDATE]: Term 1 election: Requested vote from peers c210472b61864d6ea1b62fbe69de1bd6 (127.0.98.1:40937), 8e850dac37f842b099985cda9f956954 (127.0.98.3:43295)
I20260620 14:22:58.728021 19380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712e7ccdaa8d4760badfc9c0258b17d1" candidate_uuid: "8a228d0813a546d4ba51928db89b89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e850dac37f842b099985cda9f956954"
I20260620 14:22:58.728018 19118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712e7ccdaa8d4760badfc9c0258b17d1" candidate_uuid: "8a228d0813a546d4ba51928db89b89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c210472b61864d6ea1b62fbe69de1bd6"
I20260620 14:22:58.728099 19380 raft_consensus.cc:3037] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:58.728099 19118 raft_consensus.cc:3037] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:22:58.728993 19380 raft_consensus.cc:2454] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a228d0813a546d4ba51928db89b89f0 in term 1.
I20260620 14:22:58.728996 19118 raft_consensus.cc:2454] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a228d0813a546d4ba51928db89b89f0 in term 1.
I20260620 14:22:58.729339 19183 leader_election.cc:302] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a228d0813a546d4ba51928db89b89f0, c210472b61864d6ea1b62fbe69de1bd6; no voters:
I20260620 14:22:58.729458 19447 raft_consensus.cc:2781] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:22:58.729723 19447 raft_consensus.cc:686] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8a228d0813a546d4ba51928db89b89f0, State: Running, Role: LEADER
I20260620 14:22:58.729853 19447 consensus_queue.cc:237] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } }
I20260620 14:22:58.730695 18976 catalog_manager.cc:5495] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a228d0813a546d4ba51928db89b89f0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8a228d0813a546d4ba51928db89b89f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a228d0813a546d4ba51928db89b89f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 } health_report { overall_health: UNKNOWN } } }
I20260620 14:22:58.738238 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.735773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.0.98.2:41417)
0620 14:22:58.735813 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:22:58.735815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:22:58.735824 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:22:58.735886 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:22:58.735887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:22:58.735903 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:22:58.735957 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:22:58.736052 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.736493 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.737399 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:22:58.737586 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:22:58.737652 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:22:58.737831 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:22:58.737915 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:22:58.737986 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:22:58.738044 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:22:58.738053 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:22:58.740144 19380 raft_consensus.cc:1261] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 1 FOLLOWER]: Refusing update from remote peer 8a228d0813a546d4ba51928db89b89f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:58.740157 19118 raft_consensus.cc:1261] T 712e7ccdaa8d4760badfc9c0258b17d1 P c210472b61864d6ea1b62fbe69de1bd6 [term 1 FOLLOWER]: Refusing update from remote peer 8a228d0813a546d4ba51928db89b89f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:22:58.740473 19447 consensus_queue.cc:1035] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e850dac37f842b099985cda9f956954" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:58.740614 19450 consensus_queue.cc:1035] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c210472b61864d6ea1b62fbe69de1bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:22:58.743755 19455 mvcc.cc:205] Tried to move back new op lower bound from 7298930191316770816 to 7298930191278116864. Current Snapshot: MvccSnapshot[applied={T|T < 7298930191316770816 or (T in {7298930191316770816})}]
I20260620 14:22:58.744228 19453 mvcc.cc:205] Tried to move back new op lower bound from 7298930191316770816 to 7298930191278116864. Current Snapshot: MvccSnapshot[applied={T|T < 7298930191316770816 or (T in {7298930191316770816})}]
I20260620 14:22:58.745251 18976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:58.745376 18976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:22:58.746124 18976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a1173ad3aac4552ba9bfa90caa74fc7: Sending DeleteTablet for 3 replicas of tablet 712e7ccdaa8d4760badfc9c0258b17d1
I20260620 14:22:58.746459 19229 tablet_service.cc:1514] Processing DeleteTablet for tablet 712e7ccdaa8d4760badfc9c0258b17d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:58 UTC) from {username='slave'} at 127.0.0.1:58408
I20260620 14:22:58.746753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 19036
I20260620 14:22:58.746819 19098 tablet_service.cc:1514] Processing DeleteTablet for tablet 712e7ccdaa8d4760badfc9c0258b17d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:58 UTC) from {username='slave'} at 127.0.0.1:36316
I20260620 14:22:58.746821 19360 tablet_service.cc:1514] Processing DeleteTablet for tablet 712e7ccdaa8d4760badfc9c0258b17d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:22:58 UTC) from {username='slave'} at 127.0.0.1:47154
I20260620 14:22:58.747316 19462 tablet_replica.cc:330] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: stopping tablet replica
I20260620 14:22:58.747404 19462 raft_consensus.cc:2229] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:22:58.747486 19462 raft_consensus.cc:2258] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:58.747942 19463 tablet_replica.cc:330] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: stopping tablet replica
I20260620 14:22:58.747983 19462 ts_tablet_manager.cc:1905] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:58.748138 19463 raft_consensus.cc:2229] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:22:58.748283 19463 raft_consensus.cc:2258] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:22:58.748768 19463 ts_tablet_manager.cc:1905] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:22:58.749644 19462 ts_tablet_manager.cc:1918] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:58.749722 19462 log.cc:1196] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/712e7ccdaa8d4760badfc9c0258b17d1
I20260620 14:22:58.749979 19462 ts_tablet_manager.cc:1939] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8a228d0813a546d4ba51928db89b89f0: Deleting consensus metadata
I20260620 14:22:58.750864 19463 ts_tablet_manager.cc:1918] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:22:58.750998 19463 log.cc:1196] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/712e7ccdaa8d4760badfc9c0258b17d1
I20260620 14:22:58.751010 18960 catalog_manager.cc:4841] TS 8a228d0813a546d4ba51928db89b89f0 (127.0.98.2:41417): tablet 712e7ccdaa8d4760badfc9c0258b17d1 (table test-table [id=b25046cd3d8e407397ade5945e0136a7]) successfully deleted
I20260620 14:22:58.751340 19463 ts_tablet_manager.cc:1939] T 712e7ccdaa8d4760badfc9c0258b17d1 P 8e850dac37f842b099985cda9f956954: Deleting consensus metadata
I20260620 14:22:58.751906 18961 catalog_manager.cc:4841] TS 8e850dac37f842b099985cda9f956954 (127.0.98.3:43295): tablet 712e7ccdaa8d4760badfc9c0258b17d1 (table test-table [id=b25046cd3d8e407397ade5945e0136a7]) successfully deleted
W20260620 14:22:58.754392 18961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40937 (error 108)
I20260620 14:22:58.754909 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:22:58.754766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40937 (local address 127.0.0.1:36324)
0620 14:22:58.754825 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:22:58.754871 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:22:58.754918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 19167
W20260620 14:22:58.757225 18961 catalog_manager.cc:4584] TS c210472b61864d6ea1b62fbe69de1bd6 (127.0.98.1:40937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 712e7ccdaa8d4760badfc9c0258b17d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40937: connect: Connection refused (error 111)
I20260620 14:22:58.760692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 19298
I20260620 14:22:58.766063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 18945
2026-06-20T14:22:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (670 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:22:58 dist-test-slave-h6n5 krb5kdc[19467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:22:58 dist-test-slave-h6n5 krb5kdc[19467](info): set up 2 sockets
Jun 20 14:22:58 dist-test-slave-h6n5 krb5kdc[19467](info): commencing operation
krb5kdc: starting...
W20260620 14:22:59.799489 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:22:59 dist-test-slave-h6n5 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:22:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:22:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:22:59.918504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:22:59.999905 19480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:00.000116 19480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:00.000139 19480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:00.001385 19480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:00.001441 19480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:00.001456 19480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:00.001469 19480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:00.001482 19480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:00.003046 19480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:00.003338 19480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:00.003609 19480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:00.008749 19488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.008890 19480 server_base.cc:1034] running on GCE node
W20260620 14:23:00.008759 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:00.008759 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.009277 19480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:00.009568 19480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:00.010717 19480 hybrid_clock.cc:648] HybridClock initialized: now 1781965380010693 us; error 39 us; skew 500 ppm
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:00.013435 19480 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:00.014019 19480 webserver.cc:460] Webserver started at http://127.0.98.62:32895/ using document root <none> and password file <none>
I20260620 14:23:00.014276 19480 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:00.014348 19480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:00.014490 19480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:00.015630 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "fc99459826b6447c8ea2f6194d87c851"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "f95e60e5abb183ba4a90731f2ebc9d8b"
server_key_iv: "e16c40716438df4a23f1161fe1fb3e4a"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.016004 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc99459826b6447c8ea2f6194d87c851"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "f95e60e5abb183ba4a90731f2ebc9d8b"
server_key_iv: "e16c40716438df4a23f1161fe1fb3e4a"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.017375 19480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:00.018085 19495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.018301 19480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:00.018410 19480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "fc99459826b6447c8ea2f6194d87c851"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "f95e60e5abb183ba4a90731f2ebc9d8b"
server_key_iv: "e16c40716438df4a23f1161fe1fb3e4a"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.018491 19480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:00.041003 19480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:00.043030 19480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:00.043196 19480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:00.047391 19480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38815
I20260620 14:23:00.047442 19547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38815 every 8 connection(s)
I20260620 14:23:00.047821 19480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:00.048691 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:00.050985 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851: Bootstrap starting.
I20260620 14:23:00.051627 19548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:00.051928 19548 log.cc:824] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:00.052597 19548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851: No bootstrap required, opened a new log
I20260620 14:23:00.053859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 19480
I20260620 14:23:00.053989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:00.054180 392 external_mini_cluster.cc:1428] Setting key d3744acf819ba99060ba59350496b7a1
I20260620 14:23:00.054329 19548 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc99459826b6447c8ea2f6194d87c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38815 } }
I20260620 14:23:00.054528 19548 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:00.054580 19548 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc99459826b6447c8ea2f6194d87c851, State: Initialized, Role: FOLLOWER
I20260620 14:23:00.054955 19548 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc99459826b6447c8ea2f6194d87c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38815 } }
I20260620 14:23:00.055047 19548 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:00.055079 19548 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:00.055124 19548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:00.056402 19548 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc99459826b6447c8ea2f6194d87c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38815 } }
I20260620 14:23:00.056494 19548 leader_election.cc:302] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc99459826b6447c8ea2f6194d87c851; no voters:
I20260620 14:23:00.056648 19548 leader_election.cc:288] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:00.056706 19553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:00.056865 19553 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [term 1 LEADER]: Becoming Leader. State: Replica: fc99459826b6447c8ea2f6194d87c851, State: Running, Role: LEADER
I20260620 14:23:00.056917 19548 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:00.056954 19553 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc99459826b6447c8ea2f6194d87c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38815 } }
I20260620 14:23:00.057318 19553 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc99459826b6447c8ea2f6194d87c851. Latest consensus state: current_term: 1 leader_uuid: "fc99459826b6447c8ea2f6194d87c851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc99459826b6447c8ea2f6194d87c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38815 } } }
I20260620 14:23:00.057396 19553 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:00.057330 19554 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc99459826b6447c8ea2f6194d87c851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc99459826b6447c8ea2f6194d87c851" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38815 } } }
I20260620 14:23:00.057430 19554 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:00.057793 19560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:00.058275 19560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:00.063532 19560 catalog_manager.cc:1347] Generated new cluster ID: 7b2e468c23de49aea74d10c6682e9d74
I20260620 14:23:00.063618 19560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:00.066847 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.054710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.98.62:38815)
0620 14:23:00.054886 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.054896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.054927 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:23:00.055039 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.055046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.055102 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.055259 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:00.056286 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.057731 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.058979 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.059499 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.059656 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.062363 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:00.062367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:00.062368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:00.062381 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:00.065160 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.065572 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.065575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.065575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.065695 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.065904 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.065905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.065905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.066223 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:00.066371 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.066486 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.066626 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260620 14:23:00.070546 19560 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:00.071244 19560 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:00.087353 19560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851: Generated new TSK 0
I20260620 14:23:00.087589 19560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:23:00.118662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38815
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:00.206682 19574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:00.206957 19574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:00.206987 19574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:00.208192 19574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:00.208230 19574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:00.208256 19574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:23:00.209668 19574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:00.209841 19574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:00.210022 19574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:00.214949 19580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:00.214967 19582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:00.214970 19579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.215143 19574 server_base.cc:1034] running on GCE node
I20260620 14:23:00.215387 19574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:00.215668 19574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:00.216831 19574 hybrid_clock.cc:648] HybridClock initialized: now 1781965380216817 us; error 28 us; skew 500 ppm
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:00.219342 19574 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:23:00.219820 19574 webserver.cc:460] Webserver started at http://127.0.98.1:45747/ using document root <none> and password file <none>
I20260620 14:23:00.220041 19574 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:00.220103 19574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:00.220242 19574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:00.221408 19574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "2b0ebba7864042a9990cb2a74a74241a"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "976d6c56eb56f4ad03239eb07cb2e355"
server_key_iv: "edefb184180419c9f2be28cbc8e035d6"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.221762 19574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2b0ebba7864042a9990cb2a74a74241a"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "976d6c56eb56f4ad03239eb07cb2e355"
server_key_iv: "edefb184180419c9f2be28cbc8e035d6"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.223078 19574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:00.223873 19589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.224038 19574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:00.224150 19574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "2b0ebba7864042a9990cb2a74a74241a"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "976d6c56eb56f4ad03239eb07cb2e355"
server_key_iv: "edefb184180419c9f2be28cbc8e035d6"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.224222 19574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:00.230675 19574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:00.232448 19574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:00.232599 19574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:00.232795 19574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:00.233078 19574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:00.233136 19574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.233176 19574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:00.233211 19574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.238735 19574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40909
I20260620 14:23:00.238828 19702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40909 every 8 connection(s)
I20260620 14:23:00.239114 19574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:23:00.243811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 19574
I20260620 14:23:00.243937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:23:00.244185 392 external_mini_cluster.cc:1428] Setting key bd47467cc17cde872909b49a5698c97f
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:00.249343 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.239865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57465 (local address 127.0.98.62:38815)
0620 14:23:00.239913 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.239916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.240128 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:23:00.241060 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.241061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.241115 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.241188 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:00.241782 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.242373 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.243361 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.243650 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.243718 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.246254 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:00.246256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:00.246256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:00.246264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:00.247953 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.248334 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.248335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.248336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.248401 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.248638 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.248640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.248640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.248806 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:00.248881 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.249124 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.249218 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:23:00.249361 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.239847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38815 (local address 127.0.98.1:57465)
0620 14:23:00.240013 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.240049 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.240940 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.241252 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.241256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.241497 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:00.241702 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.241706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.242455 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.242455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.243263 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.243265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.243758 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.243759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.243918 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.244617 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:00.244626 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:00.246129 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:00.248028 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:00.248034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:00.248036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:00.248256 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:00.248470 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:00.248470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:00.248470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:00.248565 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:00.248876 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:00.248881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:00.248997 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.249079 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.249195 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260620 14:23:00.249917 19703 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38815
I20260620 14:23:00.250021 19703 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:00.250203 19703 heartbeater.cc:511] Master 127.0.98.62:38815 requested a full tablet report, sending...
I20260620 14:23:00.250819 19512 ts_manager.cc:194] Registered new tserver with Master: 2b0ebba7864042a9990cb2a74a74241a (127.0.98.1:40909)
I20260620 14:23:00.251829 19512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57465
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:23:00.268529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38815
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:00.349496 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:00.349686 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:00.349709 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:00.350770 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:00.350804 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:00.350829 19708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:23:00.352295 19708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:00.352524 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:00.352756 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:00.357738 19716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:00.357740 19713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:00.357709 19714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.357951 19708 server_base.cc:1034] running on GCE node
I20260620 14:23:00.358177 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:00.358418 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:00.359591 19708 hybrid_clock.cc:648] HybridClock initialized: now 1781965380359573 us; error 32 us; skew 500 ppm
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:00.362012 19708 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:23:00.362473 19708 webserver.cc:460] Webserver started at http://127.0.98.2:45239/ using document root <none> and password file <none>
I20260620 14:23:00.362738 19708 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:00.362783 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:00.362870 19708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:00.363916 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "3494ea22bb2642439aad495cd4a5391f"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "b74a5409c8ed6880e481c799eedeec08"
server_key_iv: "89bde80e22c0fbdce4374d2a45181c63"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.364204 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3494ea22bb2642439aad495cd4a5391f"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "b74a5409c8ed6880e481c799eedeec08"
server_key_iv: "89bde80e22c0fbdce4374d2a45181c63"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.365470 19708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:23:00.366137 19723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.366325 19708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:00.366392 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "3494ea22bb2642439aad495cd4a5391f"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "b74a5409c8ed6880e481c799eedeec08"
server_key_iv: "89bde80e22c0fbdce4374d2a45181c63"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.366442 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:00.389106 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:00.391064 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:00.391180 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:00.391350 19708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:00.391716 19708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:00.391750 19708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.391772 19708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:00.391809 19708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.397539 19708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45505
I20260620 14:23:00.397581 19836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45505 every 8 connection(s)
I20260620 14:23:00.397883 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:23:00.403258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 19708
I20260620 14:23:00.403362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:23:00.403637 392 external_mini_cluster.cc:1428] Setting key 9d607e23e2c742aaceabedb3c4f4c622
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:00.407852 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.398586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52467 (local address 127.0.98.62:38815)
0620 14:23:00.398650 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.398652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.398848 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:23:00.399870 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.399871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.399884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.399939 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:00.400545 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.401040 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.402046 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.402253 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.402328 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.404563 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:00.404564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:00.404564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:00.404572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:00.406367 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.406817 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.406819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.406819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.406878 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.407099 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.407100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.407101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.407245 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:00.407311 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.407690 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.407780 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:23:00.408165 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.398571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38815 (local address 127.0.98.2:52467)
0620 14:23:00.398731 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.398766 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.399759 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.400012 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.400015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.400273 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:00.400452 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.400456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.401132 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.401133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.401955 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.401957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.402333 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.402335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.402459 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.402935 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:00.402943 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:00.404476 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:00.406468 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:00.406474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:00.406476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:00.406736 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:00.406942 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:00.406943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:00.406943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:00.407038 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:00.407342 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:00.407348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:00.407510 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.407656 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.407892 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260620 14:23:00.408674 19837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38815
I20260620 14:23:00.408771 19837 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:00.408958 19837 heartbeater.cc:511] Master 127.0.98.62:38815 requested a full tablet report, sending...
I20260620 14:23:00.409332 19512 ts_manager.cc:194] Registered new tserver with Master: 3494ea22bb2642439aad495cd4a5391f (127.0.98.2:45505)
I20260620 14:23:00.409700 19512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52467
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:23:00.429988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38815
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:00.509608 19842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:00.509912 19842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:00.509948 19842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:00.511086 19842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:00.511155 19842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:00.511202 19842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:23:00.512662 19842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:00.512923 19842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:00.513157 19842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:00.518016 19850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:00.517990 19847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.518185 19842 server_base.cc:1034] running on GCE node
W20260620 14:23:00.518010 19848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.518504 19842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:00.518749 19842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:00.519906 19842 hybrid_clock.cc:648] HybridClock initialized: now 1781965380519897 us; error 32 us; skew 500 ppm
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:00.522439 19842 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:23:00.522853 19842 webserver.cc:460] Webserver started at http://127.0.98.3:43667/ using document root <none> and password file <none>
I20260620 14:23:00.523077 19842 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:00.523120 19842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:00.523265 19842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:00.524374 19842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6439f1a3ac344999a276033259ad349"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "2e63a3fad8e6c23a6f7387edfcfbd1c1"
server_key_iv: "0db30ad7d0b5ac1a2c57f988330bed4e"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.524708 19842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6439f1a3ac344999a276033259ad349"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "2e63a3fad8e6c23a6f7387edfcfbd1c1"
server_key_iv: "0db30ad7d0b5ac1a2c57f988330bed4e"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.525977 19842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:23:00.526703 19857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.526906 19842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:00.527037 19842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "f6439f1a3ac344999a276033259ad349"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "2e63a3fad8e6c23a6f7387edfcfbd1c1"
server_key_iv: "0db30ad7d0b5ac1a2c57f988330bed4e"
server_key_version: "encryptionkey@0"
I20260620 14:23:00.527107 19842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:00.535625 19842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:00.537353 19842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:00.537499 19842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:00.537698 19842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:00.537990 19842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:00.538022 19842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.538058 19842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:00.538085 19842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:00.543663 19842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43479
I20260620 14:23:00.543720 19970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43479 every 8 connection(s)
I20260620 14:23:00.543995 19842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:23:00.544507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 19842
I20260620 14:23:00.544643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:23:00.544848 392 external_mini_cluster.cc:1428] Setting key 044989d0f2cce8104559adc7d6d1fbeb
Jun 20 14:23:00 dist-test-slave-h6n5 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:00.552829 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.544692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34547 (local address 127.0.98.62:38815)
0620 14:23:00.544759 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.544761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.544966 (+ 205us) server_negotiation.cc:407] Connection header received
0620 14:23:00.546123 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.546123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.546135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.546193 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:00.546736 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.547220 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.548237 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.548425 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.548482 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.550473 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:00.550474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:00.550475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:00.550481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:00.551648 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.552011 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.552012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.552013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.552058 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.552268 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.552268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.552269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.552370 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:00.552424 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.552707 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.552766 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:23:00.552992 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.544668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38815 (local address 127.0.98.3:34547)
0620 14:23:00.544832 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.544878 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.546032 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.546253 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.546256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.546490 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:00.546666 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.546670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.547309 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.547309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.548150 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.548152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.548496 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.548497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.548591 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.549030 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:00.549037 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:00.550391 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:00.551727 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:00.551732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:00.551734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:00.551947 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:00.552110 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:00.552111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:00.552111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:00.552204 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:00.552452 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:00.552458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:00.552596 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.552684 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.552806 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260620 14:23:00.553383 19971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38815
I20260620 14:23:00.553486 19971 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:00.553656 19971 heartbeater.cc:511] Master 127.0.98.62:38815 requested a full tablet report, sending...
I20260620 14:23:00.554018 19512 ts_manager.cc:194] Registered new tserver with Master: f6439f1a3ac344999a276033259ad349 (127.0.98.3:43479)
I20260620 14:23:00.554384 19512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34547
I20260620 14:23:00.556653 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:23:00.564357 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.558000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.98.62:38815)
0620 14:23:00.558061 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.558064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.558173 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:23:00.558225 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.558225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.558234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.558271 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:00.559101 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.559646 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.560363 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.560534 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.560584 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.561773 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:00.561775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:00.561775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:00.561781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:00.563275 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.563663 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.563664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.563664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.563716 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:00.563876 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:00.563877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:00.563877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:00.564004 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:00.564074 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.564168 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.564247 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:23:00.565951 19512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:23:00.567723 19512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:23:00.573947 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.569780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45505 (local address 127.0.0.1:42296)
0620 14:23:00.569938 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.569989 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.570060 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.570631 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.570632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.570634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:00.570701 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.570702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.571567 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.571568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.572786 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.572789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.573682 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.573683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.573761 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.573776 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.573839 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.573895 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":52,"threads_started":1}
I20260620 14:23:00.574127 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.569802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43479 (local address 127.0.0.1:52604)
0620 14:23:00.570022 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.570031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.570108 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.570493 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.570496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.570499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:00.570563 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.570565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.571938 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.571939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.572972 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.572974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.573830 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.573831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.573908 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.573911 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.573992 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.574036 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":61,"threads_started":1}
I20260620 14:23:00.574201 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.569872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.0.98.2:45505)
0620 14:23:00.570386 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.570390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.570398 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:23:00.570447 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.570449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.570464 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.570540 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:00.570774 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.571449 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.572864 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.573586 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.573667 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.573730 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.574078 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.574129 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":76,"threads_started":1}
I20260620 14:23:00.574892 19771 tablet_service.cc:1467] Processing CreateTablet for tablet 629049cd670f4170837dc9c5ccb564eb (DEFAULT_TABLE table=test-table [id=802c35f475a647498a1f70fde8ebc1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:00.575240 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.569802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40909 (local address 127.0.0.1:47440)
0620 14:23:00.570222 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.570232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.570292 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.570558 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.570559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.570560 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:00.570594 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.570595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.573039 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.573039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.573757 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.573758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.575002 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.575003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.575055 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.575058 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.575129 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.575176 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":11,"thread_start_us":56,"threads_started":1}
I20260620 14:23:00.575282 19771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629049cd670f4170837dc9c5ccb564eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:00.575294 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.570034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.0.98.1:40909)
0620 14:23:00.570229 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.570233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.570306 (+ 73us) server_negotiation.cc:407] Connection header received
0620 14:23:00.570353 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.570355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.570371 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.570471 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:00.570677 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.571838 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.574229 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.574921 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.575001 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.575054 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.575142 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.575183 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260620 14:23:00.576107 19637 tablet_service.cc:1467] Processing CreateTablet for tablet 629049cd670f4170837dc9c5ccb564eb (DEFAULT_TABLE table=test-table [id=802c35f475a647498a1f70fde8ebc1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:00.576447 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629049cd670f4170837dc9c5ccb564eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:00.578748 19985 tablet_bootstrap.cc:492] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Bootstrap starting.
I20260620 14:23:00.574245 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.570036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.0.98.3:43479)
0620 14:23:00.570320 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.570324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.570333 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:23:00.570368 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.570369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.570383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.570642 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:00.570736 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.571832 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.573039 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.573695 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.573821 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.573881 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.574010 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.574157 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
I20260620 14:23:00.579393 19985 tablet_bootstrap.cc:654] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:00.579538 19905 tablet_service.cc:1467] Processing CreateTablet for tablet 629049cd670f4170837dc9c5ccb564eb (DEFAULT_TABLE table=test-table [id=802c35f475a647498a1f70fde8ebc1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:00.579891 19905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629049cd670f4170837dc9c5ccb564eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:00.580070 19985 log.cc:824] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:00.580745 19985 tablet_bootstrap.cc:492] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: No bootstrap required, opened a new log
I20260620 14:23:00.580798 19985 ts_tablet_manager.cc:1397] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:00.582329 19988 tablet_bootstrap.cc:492] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Bootstrap starting.
I20260620 14:23:00.582291 19985 raft_consensus.cc:348] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.582412 19986 tablet_bootstrap.cc:492] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: Bootstrap starting.
I20260620 14:23:00.582482 19985 raft_consensus.cc:374] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:00.582526 19985 raft_consensus.cc:729] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3494ea22bb2642439aad495cd4a5391f, State: Initialized, Role: FOLLOWER
I20260620 14:23:00.582645 19985 consensus_queue.cc:260] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.582940 19837 heartbeater.cc:503] Master 127.0.98.62:38815 was elected leader, sending a full tablet report...
I20260620 14:23:00.582947 19985 ts_tablet_manager.cc:1428] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:23:00.583063 19986 tablet_bootstrap.cc:654] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:00.583482 19988 tablet_bootstrap.cc:654] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:00.583647 19986 log.cc:824] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:00.584144 19988 log.cc:824] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:00.584486 19986 tablet_bootstrap.cc:492] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: No bootstrap required, opened a new log
I20260620 14:23:00.584561 19986 ts_tablet_manager.cc:1397] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:23:00.584859 19988 tablet_bootstrap.cc:492] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: No bootstrap required, opened a new log
I20260620 14:23:00.584909 19988 ts_tablet_manager.cc:1397] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:23:00.586094 19986 raft_consensus.cc:348] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.586241 19986 raft_consensus.cc:374] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:00.586287 19986 raft_consensus.cc:729] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0ebba7864042a9990cb2a74a74241a, State: Initialized, Role: FOLLOWER
I20260620 14:23:00.586392 19986 consensus_queue.cc:260] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.586440 19988 raft_consensus.cc:348] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.586602 19988 raft_consensus.cc:374] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:00.586648 19988 raft_consensus.cc:729] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6439f1a3ac344999a276033259ad349, State: Initialized, Role: FOLLOWER
I20260620 14:23:00.586673 19986 ts_tablet_manager.cc:1428] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:23:00.586723 19703 heartbeater.cc:503] Master 127.0.98.62:38815 was elected leader, sending a full tablet report...
I20260620 14:23:00.586761 19988 consensus_queue.cc:260] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.587124 19988 ts_tablet_manager.cc:1428] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:23:00.587164 19971 heartbeater.cc:503] Master 127.0.98.62:38815 was elected leader, sending a full tablet report...
W20260620 14:23:00.648394 19838 tablet.cc:2369] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:23:00.739836 19704 tablet.cc:2369] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:23:00.794551 19972 tablet.cc:2369] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:23:00.831694 19993 raft_consensus.cc:482] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:00.831826 19993 raft_consensus.cc:504] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.832232 19993 leader_election.cc:288] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3494ea22bb2642439aad495cd4a5391f (127.0.98.2:45505), 2b0ebba7864042a9990cb2a74a74241a (127.0.98.1:40909)
I20260620 14:23:00.835116 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.832507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34773 (local address 127.0.98.2:45505)
0620 14:23:00.832548 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.832550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.832558 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:23:00.832591 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.832592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.832604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.832688 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:00.832858 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.833405 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.834398 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.834781 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.834834 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.834843 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.835002 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.835046 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:23:00.835122 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.832423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45505 (local address 127.0.98.3:34773)
0620 14:23:00.832461 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.832473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.832517 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.832707 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.832708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.832710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:00.832784 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.832786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.833481 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.833482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.834311 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.834314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.834851 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.834852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.834941 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.834944 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.834985 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.835036 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":44}
I20260620 14:23:00.835142 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.832496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36353 (local address 127.0.98.1:40909)
0620 14:23:00.832540 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.832542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.832692 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:23:00.832761 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.832762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.832775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.832837 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:00.833000 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.833497 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.834444 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.834798 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.834882 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.834891 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.835043 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.835080 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:23:00.835153 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.832430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40909 (local address 127.0.98.3:36353)
0620 14:23:00.832616 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.832626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:00.832686 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:00.832857 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:00.832857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:00.832859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:00.832927 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.832929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.833579 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.833580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.834338 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:00.834341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.834903 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.834904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:00.834957 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.834960 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:23:00.835030 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:23:00.835065 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":41,"thread_start_us":92,"threads_started":1}
I20260620 14:23:00.835462 19657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629049cd670f4170837dc9c5ccb564eb" candidate_uuid: "f6439f1a3ac344999a276033259ad349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0ebba7864042a9990cb2a74a74241a" is_pre_election: true
I20260620 14:23:00.835464 19791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629049cd670f4170837dc9c5ccb564eb" candidate_uuid: "f6439f1a3ac344999a276033259ad349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3494ea22bb2642439aad495cd4a5391f" is_pre_election: true
I20260620 14:23:00.835633 19657 raft_consensus.cc:2454] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6439f1a3ac344999a276033259ad349 in term 0.
I20260620 14:23:00.835633 19791 raft_consensus.cc:2454] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6439f1a3ac344999a276033259ad349 in term 0.
I20260620 14:23:00.835852 19860 leader_election.cc:302] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0ebba7864042a9990cb2a74a74241a, f6439f1a3ac344999a276033259ad349; no voters:
I20260620 14:23:00.836040 19993 raft_consensus.cc:2781] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:23:00.836079 19993 raft_consensus.cc:482] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:23:00.836094 19993 raft_consensus.cc:3037] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:00.836758 19993 raft_consensus.cc:504] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.836890 19993 leader_election.cc:288] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [CANDIDATE]: Term 1 election: Requested vote from peers 3494ea22bb2642439aad495cd4a5391f (127.0.98.2:45505), 2b0ebba7864042a9990cb2a74a74241a (127.0.98.1:40909)
I20260620 14:23:00.837047 19791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629049cd670f4170837dc9c5ccb564eb" candidate_uuid: "f6439f1a3ac344999a276033259ad349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3494ea22bb2642439aad495cd4a5391f"
I20260620 14:23:00.837070 19657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629049cd670f4170837dc9c5ccb564eb" candidate_uuid: "f6439f1a3ac344999a276033259ad349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0ebba7864042a9990cb2a74a74241a"
I20260620 14:23:00.837122 19791 raft_consensus.cc:3037] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:00.837145 19657 raft_consensus.cc:3037] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:00.837785 19657 raft_consensus.cc:2454] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6439f1a3ac344999a276033259ad349 in term 1.
I20260620 14:23:00.837803 19791 raft_consensus.cc:2454] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6439f1a3ac344999a276033259ad349 in term 1.
I20260620 14:23:00.837963 19860 leader_election.cc:302] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0ebba7864042a9990cb2a74a74241a, f6439f1a3ac344999a276033259ad349; no voters:
I20260620 14:23:00.838069 19993 raft_consensus.cc:2781] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:00.838207 19993 raft_consensus.cc:686] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 1 LEADER]: Becoming Leader. State: Replica: f6439f1a3ac344999a276033259ad349, State: Running, Role: LEADER
I20260620 14:23:00.838311 19993 consensus_queue.cc:237] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } }
I20260620 14:23:00.839108 19511 catalog_manager.cc:5495] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6439f1a3ac344999a276033259ad349 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f6439f1a3ac344999a276033259ad349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6439f1a3ac344999a276033259ad349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } health_report { overall_health: UNKNOWN } } }
I20260620 14:23:00.855522 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.852981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.0.98.3:43479)
0620 14:23:00.853022 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:00.853024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:00.853151 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:23:00.853249 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:00.853250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:00.853265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:00.853338 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:00.853499 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.853999 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.854754 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:00.854944 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:00.855016 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:00.855192 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:00.855285 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:00.855370 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:00.855399 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:23:00.855417 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:23:00.857487 19791 raft_consensus.cc:1261] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 1 FOLLOWER]: Refusing update from remote peer f6439f1a3ac344999a276033259ad349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:00.857512 19657 raft_consensus.cc:1261] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 1 FOLLOWER]: Refusing update from remote peer f6439f1a3ac344999a276033259ad349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:00.857923 19995 consensus_queue.cc:1035] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0ebba7864042a9990cb2a74a74241a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:00.858054 19993 consensus_queue.cc:1035] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3494ea22bb2642439aad495cd4a5391f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:00.860378 19998 mvcc.cc:205] Tried to move back new op lower bound from 7298930199989391360 to 7298930199914332160. Current Snapshot: MvccSnapshot[applied={T|T < 7298930199989391360}]
I20260620 14:23:00.860898 20000 mvcc.cc:205] Tried to move back new op lower bound from 7298930199989391360 to 7298930199914332160. Current Snapshot: MvccSnapshot[applied={T|T < 7298930199989391360}]
I20260620 14:23:00.860951 19999 mvcc.cc:205] Tried to move back new op lower bound from 7298930199989391360 to 7298930199914332160. Current Snapshot: MvccSnapshot[applied={T|T < 7298930199989391360 or (T in {7298930199989391360})}]
I20260620 14:23:00.862807 19511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:00.862886 19511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:00.863756 19511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc99459826b6447c8ea2f6194d87c851: Sending DeleteTablet for 3 replicas of tablet 629049cd670f4170837dc9c5ccb564eb
I20260620 14:23:00.864084 19637 tablet_service.cc:1514] Processing DeleteTablet for tablet 629049cd670f4170837dc9c5ccb564eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47440
I20260620 14:23:00.864084 19905 tablet_service.cc:1514] Processing DeleteTablet for tablet 629049cd670f4170837dc9c5ccb564eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52604
I20260620 14:23:00.864097 19771 tablet_service.cc:1514] Processing DeleteTablet for tablet 629049cd670f4170837dc9c5ccb564eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42296
I20260620 14:23:00.864301 20008 tablet_replica.cc:330] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: stopping tablet replica
I20260620 14:23:00.864396 20008 raft_consensus.cc:2229] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:23:00.864506 20010 tablet_replica.cc:330] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: stopping tablet replica
I20260620 14:23:00.864535 20008 raft_consensus.cc:2258] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:00.864581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 19574
I20260620 14:23:00.864689 20009 tablet_replica.cc:330] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a: stopping tablet replica
I20260620 14:23:00.864789 20009 raft_consensus.cc:2229] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:00.864892 20009 raft_consensus.cc:2258] T 629049cd670f4170837dc9c5ccb564eb P 2b0ebba7864042a9990cb2a74a74241a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:00.865170 20008 ts_tablet_manager.cc:1905] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:00.866372 20010 raft_consensus.cc:2229] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:00.866518 20010 raft_consensus.cc:2258] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:00.866696 20008 ts_tablet_manager.cc:1918] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:00.866763 20008 log.cc:1196] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/629049cd670f4170837dc9c5ccb564eb
I20260620 14:23:00.867050 20008 ts_tablet_manager.cc:1939] T 629049cd670f4170837dc9c5ccb564eb P f6439f1a3ac344999a276033259ad349: Deleting consensus metadata
I20260620 14:23:00.867156 20010 ts_tablet_manager.cc:1905] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:00.867431 19497 catalog_manager.cc:4841] TS f6439f1a3ac344999a276033259ad349 (127.0.98.3:43479): tablet 629049cd670f4170837dc9c5ccb564eb (table test-table [id=802c35f475a647498a1f70fde8ebc1de]) successfully deleted
I20260620 14:23:00.869144 20010 ts_tablet_manager.cc:1918] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:00.869220 20010 log.cc:1196] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/629049cd670f4170837dc9c5ccb564eb
I20260620 14:23:00.869451 20010 ts_tablet_manager.cc:1939] T 629049cd670f4170837dc9c5ccb564eb P 3494ea22bb2642439aad495cd4a5391f: Deleting consensus metadata
W20260620 14:23:00.871650 19498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40909 (error 108)
I20260620 14:23:00.871781 19496 catalog_manager.cc:4841] TS 3494ea22bb2642439aad495cd4a5391f (127.0.98.2:45505): tablet 629049cd670f4170837dc9c5ccb564eb (table test-table [id=802c35f475a647498a1f70fde8ebc1de]) successfully deleted
I20260620 14:23:00.872265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 19708
I20260620 14:23:00.872292 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:00.872153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40909 (local address 127.0.0.1:47442)
0620 14:23:00.872229 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:00.872265 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:23:00.872419 19498 catalog_manager.cc:4584] TS 2b0ebba7864042a9990cb2a74a74241a (127.0.98.1:40909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 629049cd670f4170837dc9c5ccb564eb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40909: connect: Connection refused (error 111)
I20260620 14:23:00.878703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 19842
I20260620 14:23:00.883337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 19480
2026-06-20T14:23:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2117 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:00Z Disabled control of system clock
I20260620 14:23:00.909775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36521
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:23:00.989677 20015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:00.989883 20015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:00.989903 20015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:00.991042 20015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:00.991084 20015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:00.991099 20015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:00.991137 20015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:00.991168 20015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:00.992506 20015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:00.992744 20015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:00.993036 20015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:00.997851 20023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.997915 20015 server_base.cc:1034] running on GCE node
W20260620 14:23:00.997877 20021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:00.997877 20020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:00.998266 20015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:00.998510 20015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:00.999616 20015 hybrid_clock.cc:648] HybridClock initialized: now 1781965380999595 us; error 32 us; skew 500 ppm
I20260620 14:23:01.000808 20015 webserver.cc:460] Webserver started at http://127.0.98.62:43233/ using document root <none> and password file <none>
I20260620 14:23:01.001057 20015 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:01.001106 20015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:01.001273 20015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:01.002398 20015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "89d9edeb311a4cd2968c408e779c1314"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "efc209317d65d915c9db813e2cdece16"
server_key_iv: "1e9dba823bdfa33481b3960721566b26"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.002753 20015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "89d9edeb311a4cd2968c408e779c1314"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "efc209317d65d915c9db813e2cdece16"
server_key_iv: "1e9dba823bdfa33481b3960721566b26"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.004117 20015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:23:01.004932 20029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.005126 20015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:01.005244 20015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "89d9edeb311a4cd2968c408e779c1314"
format_stamp: "Formatted at 2026-06-20 14:23:00 on dist-test-slave-h6n5"
server_key: "efc209317d65d915c9db813e2cdece16"
server_key_iv: "1e9dba823bdfa33481b3960721566b26"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.005316 20015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:01.031023 20015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:01.031306 20015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:01.031420 20015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:01.035287 20015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36521
I20260620 14:23:01.035337 20081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36521 every 8 connection(s)
I20260620 14:23:01.035712 20015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:01.036733 20082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:01.039013 20082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314: Bootstrap starting.
I20260620 14:23:01.039661 20082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:01.039959 20082 log.cc:824] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:01.040642 20082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314: No bootstrap required, opened a new log
I20260620 14:23:01.042273 20082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9edeb311a4cd2968c408e779c1314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260620 14:23:01.042430 20082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:01.042475 20082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d9edeb311a4cd2968c408e779c1314, State: Initialized, Role: FOLLOWER
I20260620 14:23:01.042614 20082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9edeb311a4cd2968c408e779c1314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260620 14:23:01.042692 20082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:01.042737 20082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:01.042783 20082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:01.043301 20082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9edeb311a4cd2968c408e779c1314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260620 14:23:01.043414 20082 leader_election.cc:302] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89d9edeb311a4cd2968c408e779c1314; no voters:
I20260620 14:23:01.043622 20082 leader_election.cc:288] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:01.043663 20085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:01.043879 20085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [term 1 LEADER]: Becoming Leader. State: Replica: 89d9edeb311a4cd2968c408e779c1314, State: Running, Role: LEADER
I20260620 14:23:01.044006 20085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9edeb311a4cd2968c408e779c1314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260620 14:23:01.044050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20015
I20260620 14:23:01.044159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:01.044369 392 external_mini_cluster.cc:1428] Setting key c5e8231b574ff33fe3f1ab1406f4e43c
I20260620 14:23:01.044406 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89d9edeb311a4cd2968c408e779c1314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9edeb311a4cd2968c408e779c1314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } } }
I20260620 14:23:01.044505 20087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89d9edeb311a4cd2968c408e779c1314. Latest consensus state: current_term: 1 leader_uuid: "89d9edeb311a4cd2968c408e779c1314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9edeb311a4cd2968c408e779c1314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } } }
I20260620 14:23:01.044559 20082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:01.044576 20087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:01.044507 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:01.045555 20089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:01.046103 20089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:01.047638 20089 catalog_manager.cc:1347] Generated new cluster ID: 3728b35badf54a2e99c80c819d5ee512
I20260620 14:23:01.047699 20089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:01.050215 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.044835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.0.98.62:36521)
0620 14:23:01.045017 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.045031 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.045278 (+ 247us) server_negotiation.cc:407] Connection header received
0620 14:23:01.045428 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.045435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.045524 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.045672 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:01.045951 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.047389 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.048266 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.048722 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.048848 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.049563 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:01.049567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:01.049568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:01.049747 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:01.049780 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:01.049792 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:01.049876 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.049957 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.050039 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260620 14:23:01.055824 20089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:01.056376 20089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:01.059751 20089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314: Generated new TSK 0
I20260620 14:23:01.059901 20089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:01.067761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:23:01.166949 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:01.167176 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:01.167199 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:01.168444 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:01.168483 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:01.168507 20106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:23:01.169816 20106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:01.170045 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:01.170342 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:01.176106 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.176146 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.176297 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:01.176295 20106 server_base.cc:1034] running on GCE node
I20260620 14:23:01.176585 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:01.176842 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:01.178001 20106 hybrid_clock.cc:648] HybridClock initialized: now 1781965381177985 us; error 31 us; skew 500 ppm
I20260620 14:23:01.179348 20106 webserver.cc:460] Webserver started at http://127.0.98.1:37585/ using document root <none> and password file <none>
I20260620 14:23:01.179651 20106 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:01.179720 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:01.179854 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:01.180895 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data/instance:
uuid: "5243e9eb19a04319b056ad89ebb02bfb"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "424b74849aa134021771b4d12b6c91a0"
server_key_iv: "7eb70ccefb0662ea78497ce8dae3a88b"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.181269 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5243e9eb19a04319b056ad89ebb02bfb"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "424b74849aa134021771b4d12b6c91a0"
server_key_iv: "7eb70ccefb0662ea78497ce8dae3a88b"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.182639 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:01.183369 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.183594 20106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:01.183694 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/wal
uuid: "5243e9eb19a04319b056ad89ebb02bfb"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "424b74849aa134021771b4d12b6c91a0"
server_key_iv: "7eb70ccefb0662ea78497ce8dae3a88b"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.183759 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:01.203610 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:01.203943 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:01.204087 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:01.204324 20106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:01.204711 20106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:01.204761 20106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.204792 20106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:01.204808 20106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.210644 20106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35647
I20260620 14:23:01.210707 20233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35647 every 8 connection(s)
I20260620 14:23:01.211035 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:23:01.212906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20106
I20260620 14:23:01.212992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:23:01.213200 392 external_mini_cluster.cc:1428] Setting key 68615eaeb08b1e283d5b9efb0146bb8a
I20260620 14:23:01.214079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:23:01.216650 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.211797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58393 (local address 127.0.98.62:36521)
0620 14:23:01.211856 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.211857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.212107 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:23:01.212634 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.212648 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.212662 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.212718 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:01.213531 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.214024 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.215110 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.215379 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.215437 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.216381 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:01.216382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:01.216383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:01.216405 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:01.216422 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:01.216427 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:01.216489 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.216569 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.216594 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:23:01.216943 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.211811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.1:58393)
0620 14:23:01.211994 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.212027 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.212532 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.212788 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.212796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.212805 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:01.213428 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.213433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.214115 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.214116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.214994 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.214996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.215459 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.215461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.215657 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.216240 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:23:01.216241 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:01.216278 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:23:01.216306 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:01.216495 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:01.216500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:01.216503 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.216572 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.216686 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260620 14:23:01.217401 20234 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260620 14:23:01.217511 20234 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:01.217686 20234 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260620 14:23:01.218179 20046 ts_manager.cc:194] Registered new tserver with Master: 5243e9eb19a04319b056ad89ebb02bfb (127.0.98.1:35647)
I20260620 14:23:01.218760 20046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58393
W20260620 14:23:01.294292 20237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:01.294492 20237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:01.294514 20237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:01.295672 20237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:01.295710 20237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:01.295742 20237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:23:01.297052 20237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:01.297295 20237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:01.297578 20237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:01.302690 20242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.302682 20243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.302682 20245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:01.302699 20237 server_base.cc:1034] running on GCE node
I20260620 14:23:01.303097 20237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:01.303354 20237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:01.304508 20237 hybrid_clock.cc:648] HybridClock initialized: now 1781965381304488 us; error 51 us; skew 500 ppm
I20260620 14:23:01.305843 20237 webserver.cc:460] Webserver started at http://127.0.98.2:42539/ using document root <none> and password file <none>
I20260620 14:23:01.306092 20237 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:01.306162 20237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:01.306295 20237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:01.307348 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ad91cc7fdcb413c9a89dab206f74d28"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "b218a5905bc028d88c838daee855f3c1"
server_key_iv: "43039617a5f0676bc6a24559b23cf4b1"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.307749 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ad91cc7fdcb413c9a89dab206f74d28"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "b218a5905bc028d88c838daee855f3c1"
server_key_iv: "43039617a5f0676bc6a24559b23cf4b1"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.309172 20237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:01.309988 20251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.310153 20237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:01.310221 20237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal
uuid: "2ad91cc7fdcb413c9a89dab206f74d28"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "b218a5905bc028d88c838daee855f3c1"
server_key_iv: "43039617a5f0676bc6a24559b23cf4b1"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.310272 20237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:01.316557 20237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:01.316829 20237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:01.316929 20237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:01.317091 20237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:01.317349 20237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:01.317384 20237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.317404 20237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:01.317425 20237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.323060 20237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34631
I20260620 14:23:01.323100 20364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34631 every 8 connection(s)
I20260620 14:23:01.323403 20237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:23:01.328504 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.324197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50725 (local address 127.0.98.62:36521)
0620 14:23:01.324257 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.324260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.324476 (+ 216us) server_negotiation.cc:407] Connection header received
0620 14:23:01.324977 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.324978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.324989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.325073 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:01.325794 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.326282 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.327197 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.327379 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.327433 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.328279 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:01.328281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:01.328281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:01.328286 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:01.328299 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:01.328303 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:01.328347 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.328422 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.328445 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:23:01.328702 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.324128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.2:50725)
0620 14:23:01.324357 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.324391 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.324867 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.325162 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.325166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.325174 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:01.325685 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.325690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.326365 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.326366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.327125 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.327127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.327462 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.327463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.327629 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.328141 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:23:01.328141 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:01.328175 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:23:01.328202 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:01.328354 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:01.328358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:01.328360 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.328412 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.328495 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260620 14:23:01.329118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20237
I20260620 14:23:01.329283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:23:01.329275 20365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260620 14:23:01.329362 20365 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:01.329530 392 external_mini_cluster.cc:1428] Setting key 98328fba71ea02f2a6a9a784c27fd9eb
I20260620 14:23:01.329566 20365 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260620 14:23:01.329959 20045 ts_manager.cc:194] Registered new tserver with Master: 2ad91cc7fdcb413c9a89dab206f74d28 (127.0.98.2:34631)
I20260620 14:23:01.330333 20045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50725
I20260620 14:23:01.330499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--builtin_ntp_servers=127.0.98.20:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:23:01.410293 20368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:01.410521 20368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:01.410557 20368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:01.411742 20368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:01.411793 20368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:01.411834 20368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:23:01.413206 20368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:01.413458 20368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:01.413787 20368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:01.418627 20376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.418639 20373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:01.418659 20374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:01.418983 20368 server_base.cc:1034] running on GCE node
I20260620 14:23:01.419186 20368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:01.419431 20368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:01.420585 20368 hybrid_clock.cc:648] HybridClock initialized: now 1781965381420566 us; error 28 us; skew 500 ppm
I20260620 14:23:01.421737 20368 webserver.cc:460] Webserver started at http://127.0.98.3:36453/ using document root <none> and password file <none>
I20260620 14:23:01.421967 20368 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:01.422031 20368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:01.422180 20368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:01.423269 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f134a704a824d21b6c14715aa7781d2"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "b82f9ff0f05f979e07721ae4ce3e495b"
server_key_iv: "bac1e28b38010cdb660bdd1df1b4bc1b"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.423687 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f134a704a824d21b6c14715aa7781d2"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "b82f9ff0f05f979e07721ae4ce3e495b"
server_key_iv: "bac1e28b38010cdb660bdd1df1b4bc1b"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.424950 20368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:01.425668 20382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.425838 20368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:01.425932 20368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal
uuid: "0f134a704a824d21b6c14715aa7781d2"
format_stamp: "Formatted at 2026-06-20 14:23:01 on dist-test-slave-h6n5"
server_key: "b82f9ff0f05f979e07721ae4ce3e495b"
server_key_iv: "bac1e28b38010cdb660bdd1df1b4bc1b"
server_key_version: "encryptionkey@0"
I20260620 14:23:01.426011 20368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:01.435302 20368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:01.435631 20368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:01.435770 20368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:01.435969 20368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:23:01.436296 20368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:23:01.436348 20368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.436388 20368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:23:01.436415 20368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:01.442005 20368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46067
I20260620 14:23:01.442054 20495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46067 every 8 connection(s)
I20260620 14:23:01.442413 20368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:23:01.444978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20368
I20260620 14:23:01.445147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:23:01.445379 392 external_mini_cluster.cc:1428] Setting key 9205b5dada75bdb42d5830cee4146371
I20260620 14:23:01.448089 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.443171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45539 (local address 127.0.98.62:36521)
0620 14:23:01.443216 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.443218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.443460 (+ 242us) server_negotiation.cc:407] Connection header received
0620 14:23:01.444020 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.444021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.444033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.444096 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:01.444845 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.445639 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.446841 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.447027 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.447095 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.447892 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:01.447893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:01.447893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:01.447898 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:01.447911 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:01.447915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:01.447959 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.448007 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.448032 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:23:01.448328 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.443143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.3:45539)
0620 14:23:01.443318 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.443358 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.443928 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.444168 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.444172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.444180 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:01.444765 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.444772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.445741 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.445743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.446755 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.446758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.447097 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.447098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.447205 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.447769 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:23:01.447769 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:01.447796 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:23:01.447818 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:01.447959 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:01.447961 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:01.447963 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.448009 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.448080 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260620 14:23:01.448828 20496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260620 14:23:01.448923 20496 heartbeater.cc:464] Registering TS with master...
I20260620 14:23:01.449095 20496 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260620 14:23:01.449460 20045 ts_manager.cc:194] Registered new tserver with Master: 0f134a704a824d21b6c14715aa7781d2 (127.0.98.3:46067)
I20260620 14:23:01.449795 20045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45539
I20260620 14:23:01.457258 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:23:01.461448 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.458714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.98.62:36521)
0620 14:23:01.458780 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.458782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.458803 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:23:01.458821 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.458822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.458834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.458879 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:01.459050 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.459541 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.460306 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.460498 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.460579 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.461209 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:01.461211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:01.461211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:01.461216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:01.461236 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:01.461240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:01.461288 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.461341 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.461366 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:23:01.463385 20045 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:23:01.465039 20045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:23:01.471485 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.467199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46067 (local address 127.0.0.1:49434)
0620 14:23:01.467325 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.467365 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.467426 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.467921 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.467924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.467927 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:01.467998 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.468000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.469307 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.469308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.470261 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.470263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.471170 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.471171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.471257 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.471272 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.471353 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.471410 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260620 14:23:01.471494 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.467294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.0.98.3:46067)
0620 14:23:01.467451 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.467456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.467467 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:23:01.467507 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.467509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.467521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.467632 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:01.468066 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.469215 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.470343 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.471079 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.471153 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.471245 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.471374 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.471409 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260620 14:23:01.472085 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.467618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35647 (local address 127.0.0.1:52654)
0620 14:23:01.468197 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.468206 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.468260 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.468436 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.468437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.468438 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:01.468475 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.468476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.469663 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.469664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.470677 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.470680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.471508 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.471509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.471918 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.471923 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.471987 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.472022 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":61,"threads_started":1}
I20260620 14:23:01.472307 20430 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc19cf1b464cc193aad17a7888afa0 (DEFAULT_TABLE table=test-table [id=382651ca2e034d23a81f4331d4b3a868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:01.472445 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.467718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.0.98.1:35647)
0620 14:23:01.468148 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.468152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.468287 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:23:01.468335 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.468337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.468352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.468431 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:01.468529 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.469573 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.470765 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.471334 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.471394 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.471475 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.472015 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.472360 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":110,"threads_started":1}
I20260620 14:23:01.472698 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.467422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34631 (local address 127.0.0.1:57930)
0620 14:23:01.467734 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.467745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.467804 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.468013 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.468013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.468015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:01.468049 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.468050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.470463 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.470464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.471502 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.471505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.472466 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.472467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.472533 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.472537 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.472592 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.472631 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":83,"threads_started":1}
I20260620 14:23:01.472705 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abc19cf1b464cc193aad17a7888afa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:01.472973 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.467668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.0.98.2:34631)
0620 14:23:01.467856 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.467861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.467872 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:23:01.467910 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.467911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.467926 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.468005 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:01.469330 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.470385 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.471618 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.472280 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.472353 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.472430 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.472850 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.472887 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260620 14:23:01.473088 20168 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc19cf1b464cc193aad17a7888afa0 (DEFAULT_TABLE table=test-table [id=382651ca2e034d23a81f4331d4b3a868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:01.473404 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abc19cf1b464cc193aad17a7888afa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:01.473510 20299 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc19cf1b464cc193aad17a7888afa0 (DEFAULT_TABLE table=test-table [id=382651ca2e034d23a81f4331d4b3a868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:23:01.473732 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abc19cf1b464cc193aad17a7888afa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:01.476162 20510 tablet_bootstrap.cc:492] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Bootstrap starting.
I20260620 14:23:01.476662 20512 tablet_bootstrap.cc:492] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb: Bootstrap starting.
I20260620 14:23:01.476788 20511 tablet_bootstrap.cc:492] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Bootstrap starting.
I20260620 14:23:01.476928 20510 tablet_bootstrap.cc:654] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:01.477213 20510 log.cc:824] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:01.477331 20512 tablet_bootstrap.cc:654] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:01.477396 20511 tablet_bootstrap.cc:654] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:01.477675 20512 log.cc:824] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:01.477757 20511 log.cc:824] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:01.477923 20510 tablet_bootstrap.cc:492] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: No bootstrap required, opened a new log
I20260620 14:23:01.478005 20510 ts_tablet_manager.cc:1397] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:23:01.478505 20511 tablet_bootstrap.cc:492] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: No bootstrap required, opened a new log
I20260620 14:23:01.478569 20511 ts_tablet_manager.cc:1397] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:23:01.478705 20512 tablet_bootstrap.cc:492] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb: No bootstrap required, opened a new log
I20260620 14:23:01.478771 20512 ts_tablet_manager.cc:1397] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:23:01.480032 20510 raft_consensus.cc:348] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.480337 20510 raft_consensus.cc:374] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:01.480422 20510 raft_consensus.cc:729] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f134a704a824d21b6c14715aa7781d2, State: Initialized, Role: FOLLOWER
I20260620 14:23:01.480569 20510 consensus_queue.cc:260] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.480894 20510 ts_tablet_manager.cc:1428] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:01.481061 20496 heartbeater.cc:503] Master 127.0.98.62:36521 was elected leader, sending a full tablet report...
I20260620 14:23:01.481297 20512 raft_consensus.cc:348] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.481207 20511 raft_consensus.cc:348] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.481453 20512 raft_consensus.cc:374] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:01.481465 20511 raft_consensus.cc:374] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:01.481498 20512 raft_consensus.cc:729] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5243e9eb19a04319b056ad89ebb02bfb, State: Initialized, Role: FOLLOWER
I20260620 14:23:01.481500 20511 raft_consensus.cc:729] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad91cc7fdcb413c9a89dab206f74d28, State: Initialized, Role: FOLLOWER
I20260620 14:23:01.481613 20511 consensus_queue.cc:260] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.481613 20512 consensus_queue.cc:260] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.481861 20512 ts_tablet_manager.cc:1428] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:01.481861 20511 ts_tablet_manager.cc:1428] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:23:01.481917 20234 heartbeater.cc:503] Master 127.0.98.62:36521 was elected leader, sending a full tablet report...
I20260620 14:23:01.481972 20365 heartbeater.cc:503] Master 127.0.98.62:36521 was elected leader, sending a full tablet report...
I20260620 14:23:01.533149 20518 raft_consensus.cc:482] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:23:01.533298 20518 raft_consensus.cc:504] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.533725 20518 leader_election.cc:288] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f134a704a824d21b6c14715aa7781d2 (127.0.98.3:46067), 5243e9eb19a04319b056ad89ebb02bfb (127.0.98.1:35647)
I20260620 14:23:01.536815 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.534105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37511 (local address 127.0.98.1:35647)
0620 14:23:01.534152 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.534154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.534245 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:23:01.534315 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.534317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.534332 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.534389 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:01.534548 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.535015 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.536101 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.536467 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.536546 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.536557 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.536714 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.536751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:23:01.536823 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.534016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35647 (local address 127.0.98.2:37511)
0620 14:23:01.534169 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.534178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.534249 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.534392 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.534392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.534394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:01.534469 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.534471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.535096 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.535097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.535999 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.536002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.536557 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.536558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.536635 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.536638 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.536702 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.536740 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":17,"thread_start_us":102,"threads_started":1}
I20260620 14:23:01.536885 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.534016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53827 (local address 127.0.98.3:46067)
0620 14:23:01.534109 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.534112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.534122 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:23:01.534156 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.534157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.534168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.534270 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:23:01.534452 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.534974 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.536087 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.536500 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.536585 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.536595 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.536765 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.536806 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:23:01.536918 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.533903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46067 (local address 127.0.98.2:53827)
0620 14:23:01.533961 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.533975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:01.534052 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:01.534284 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:01.534285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:01.534287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:23:01.534372 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.534374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.535061 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.535062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.535990 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:01.535992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.536585 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.536586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:01.536667 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.536670 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:23:01.536763 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:23:01.536806 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":52}
I20260620 14:23:01.537281 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc19cf1b464cc193aad17a7888afa0" candidate_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f134a704a824d21b6c14715aa7781d2" is_pre_election: true
I20260620 14:23:01.537295 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc19cf1b464cc193aad17a7888afa0" candidate_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5243e9eb19a04319b056ad89ebb02bfb" is_pre_election: true
I20260620 14:23:01.537446 20450 raft_consensus.cc:2454] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad91cc7fdcb413c9a89dab206f74d28 in term 0.
I20260620 14:23:01.537446 20188 raft_consensus.cc:2454] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad91cc7fdcb413c9a89dab206f74d28 in term 0.
I20260620 14:23:01.537688 20253 leader_election.cc:302] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad91cc7fdcb413c9a89dab206f74d28, 5243e9eb19a04319b056ad89ebb02bfb; no voters:
I20260620 14:23:01.537815 20518 raft_consensus.cc:2781] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:23:01.537896 20518 raft_consensus.cc:482] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:23:01.537931 20518 raft_consensus.cc:3037] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:01.538652 20518 raft_consensus.cc:504] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.538786 20518 leader_election.cc:288] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [CANDIDATE]: Term 1 election: Requested vote from peers 0f134a704a824d21b6c14715aa7781d2 (127.0.98.3:46067), 5243e9eb19a04319b056ad89ebb02bfb (127.0.98.1:35647)
I20260620 14:23:01.538988 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc19cf1b464cc193aad17a7888afa0" candidate_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5243e9eb19a04319b056ad89ebb02bfb"
I20260620 14:23:01.538975 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc19cf1b464cc193aad17a7888afa0" candidate_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f134a704a824d21b6c14715aa7781d2"
I20260620 14:23:01.539067 20450 raft_consensus.cc:3037] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:01.539067 20188 raft_consensus.cc:3037] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:01.540076 20188 raft_consensus.cc:2454] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad91cc7fdcb413c9a89dab206f74d28 in term 1.
I20260620 14:23:01.540076 20450 raft_consensus.cc:2454] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad91cc7fdcb413c9a89dab206f74d28 in term 1.
I20260620 14:23:01.540259 20253 leader_election.cc:302] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad91cc7fdcb413c9a89dab206f74d28, 5243e9eb19a04319b056ad89ebb02bfb; no voters:
I20260620 14:23:01.540376 20518 raft_consensus.cc:2781] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:01.540553 20518 raft_consensus.cc:686] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 1 LEADER]: Becoming Leader. State: Replica: 2ad91cc7fdcb413c9a89dab206f74d28, State: Running, Role: LEADER
I20260620 14:23:01.540655 20518 consensus_queue.cc:237] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } }
I20260620 14:23:01.541594 20044 catalog_manager.cc:5495] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ad91cc7fdcb413c9a89dab206f74d28 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad91cc7fdcb413c9a89dab206f74d28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } health_report { overall_health: UNKNOWN } } }
I20260620 14:23:01.564028 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.561545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.98.2:34631)
0620 14:23:01.561619 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:01.561620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:01.561630 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:23:01.561668 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:01.561668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:01.561686 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:01.561767 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:01.561890 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.562371 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.563107 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:01.563354 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:01.563412 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:01.563653 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:01.563763 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:01.563842 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:01.563882 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:23:01.563900 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:23:01.565886 20188 raft_consensus.cc:1261] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 1 FOLLOWER]: Refusing update from remote peer 2ad91cc7fdcb413c9a89dab206f74d28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:01.565899 20450 raft_consensus.cc:1261] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 1 FOLLOWER]: Refusing update from remote peer 2ad91cc7fdcb413c9a89dab206f74d28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:01.566283 20518 consensus_queue.cc:1035] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5243e9eb19a04319b056ad89ebb02bfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:01.566488 20520 consensus_queue.cc:1035] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f134a704a824d21b6c14715aa7781d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:01.568849 20523 mvcc.cc:205] Tried to move back new op lower bound from 7298930202890919936 to 7298930202791223296. Current Snapshot: MvccSnapshot[applied={T|T < 7298930202890919936}]
I20260620 14:23:01.568992 20524 mvcc.cc:205] Tried to move back new op lower bound from 7298930202890919936 to 7298930202791223296. Current Snapshot: MvccSnapshot[applied={T|T < 7298930202890919936}]
I20260620 14:23:01.569504 20525 mvcc.cc:205] Tried to move back new op lower bound from 7298930202890919936 to 7298930202791223296. Current Snapshot: MvccSnapshot[applied={T|T < 7298930202890919936 or (T in {7298930202890919936})}]
I20260620 14:23:01.571938 20046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:01.572031 20046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:23:01.572860 20046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89d9edeb311a4cd2968c408e779c1314: Sending DeleteTablet for 3 replicas of tablet 2abc19cf1b464cc193aad17a7888afa0
I20260620 14:23:01.573211 20299 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abc19cf1b464cc193aad17a7888afa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:01 UTC) from {username='slave'} at 127.0.0.1:57930
I20260620 14:23:01.573211 20430 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abc19cf1b464cc193aad17a7888afa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:01 UTC) from {username='slave'} at 127.0.0.1:49434
I20260620 14:23:01.573283 20168 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abc19cf1b464cc193aad17a7888afa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:23:01 UTC) from {username='slave'} at 127.0.0.1:52654
I20260620 14:23:01.573467 20534 tablet_replica.cc:330] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: stopping tablet replica
I20260620 14:23:01.573549 20535 tablet_replica.cc:330] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb: stopping tablet replica
I20260620 14:23:01.573647 20534 raft_consensus.cc:2229] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:23:01.573673 20535 raft_consensus.cc:2229] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:01.573776 20535 raft_consensus.cc:2258] T 2abc19cf1b464cc193aad17a7888afa0 P 5243e9eb19a04319b056ad89ebb02bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:01.573786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20106
I20260620 14:23:01.574527 20534 raft_consensus.cc:2258] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:01.574813 20533 tablet_replica.cc:330] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: stopping tablet replica
I20260620 14:23:01.574941 20533 raft_consensus.cc:2229] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:23:01.575042 20533 raft_consensus.cc:2258] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:23:01.575423 20534 ts_tablet_manager.cc:1905] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:01.576381 20533 ts_tablet_manager.cc:1905] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:23:01.577407 20534 ts_tablet_manager.cc:1918] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:01.577488 20534 log.cc:1196] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-1/wal/wals/2abc19cf1b464cc193aad17a7888afa0
I20260620 14:23:01.577760 20534 ts_tablet_manager.cc:1939] T 2abc19cf1b464cc193aad17a7888afa0 P 2ad91cc7fdcb413c9a89dab206f74d28: Deleting consensus metadata
I20260620 14:23:01.578215 20031 catalog_manager.cc:4841] TS 2ad91cc7fdcb413c9a89dab206f74d28 (127.0.98.2:34631): tablet 2abc19cf1b464cc193aad17a7888afa0 (table test-table [id=382651ca2e034d23a81f4331d4b3a868]) successfully deleted
I20260620 14:23:01.578274 20533 ts_tablet_manager.cc:1918] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:23:01.578346 20533 log.cc:1196] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Deleting WAL directory at /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965139717380-392-0/minicluster-data/ts-2/wal/wals/2abc19cf1b464cc193aad17a7888afa0
I20260620 14:23:01.578615 20533 ts_tablet_manager.cc:1939] T 2abc19cf1b464cc193aad17a7888afa0 P 0f134a704a824d21b6c14715aa7781d2: Deleting consensus metadata
I20260620 14:23:01.579046 20031 catalog_manager.cc:4841] TS 0f134a704a824d21b6c14715aa7781d2 (127.0.98.3:46067): tablet 2abc19cf1b464cc193aad17a7888afa0 (table test-table [id=382651ca2e034d23a81f4331d4b3a868]) successfully deleted
W20260620 14:23:01.579620 20031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35647 (error 108)
I20260620 14:23:01.580171 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:01.580008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35647 (local address 127.0.0.1:52656)
0620 14:23:01.580066 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:01.580123 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:23:01.580161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20237
W20260620 14:23:01.582255 20031 catalog_manager.cc:4584] TS 5243e9eb19a04319b056ad89ebb02bfb (127.0.98.1:35647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2abc19cf1b464cc193aad17a7888afa0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35647: connect: Connection refused (error 111)
I20260620 14:23:01.585785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20368
I20260620 14:23:01.590914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20015
2026-06-20T14:23:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (707 ms)
[----------] 6 tests from EncryptionPolicyTest (8566 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:01 dist-test-slave-h6n5 krb5kdc[20539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:01 dist-test-slave-h6n5 krb5kdc[20539](info): set up 2 sockets
Jun 20 14:23:01 dist-test-slave-h6n5 krb5kdc[20539](info): commencing operation
krb5kdc: starting...
W20260620 14:23:02.627684 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:02 dist-test-slave-h6n5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:02.744773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:02.823473 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:02.823729 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:02.823768 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:02.824926 20552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:02.824991 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:02.825018 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:02.825048 20552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:02.825078 20552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:02.826541 20552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:02.826807 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:02.827020 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:23:02.831899 20552 server_base.cc:1034] running on GCE node
W20260620 14:23:02.831884 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:02.831884 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:02.831882 20557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:02.832314 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:02.832607 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:02.833781 20552 hybrid_clock.cc:648] HybridClock initialized: now 1781965382833736 us; error 53 us; skew 500 ppm
Jun 20 14:23:02 dist-test-slave-h6n5 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:02.836750 20552 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:02.837231 20552 webserver.cc:460] Webserver started at http://127.0.0.1:44015/ using document root <none> and password file <none>
I20260620 14:23:02.837451 20552 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:02.837496 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:02.837639 20552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:02.838667 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "c27c91913e324b6e8e81801c846ad5c0"
format_stamp: "Formatted at 2026-06-20 14:23:02 on dist-test-slave-h6n5"
server_key: "2600d3430e663c6f67ec06264abd8da9"
server_key_iv: "4ceab0bfdf987f2d1c718608e69d11bf"
server_key_version: "encryptionkey@0"
I20260620 14:23:02.839030 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "c27c91913e324b6e8e81801c846ad5c0"
format_stamp: "Formatted at 2026-06-20 14:23:02 on dist-test-slave-h6n5"
server_key: "2600d3430e663c6f67ec06264abd8da9"
server_key_iv: "4ceab0bfdf987f2d1c718608e69d11bf"
server_key_version: "encryptionkey@0"
I20260620 14:23:02.840409 20552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:23:02.841171 20567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:02.841342 20552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:02.841439 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "c27c91913e324b6e8e81801c846ad5c0"
format_stamp: "Formatted at 2026-06-20 14:23:02 on dist-test-slave-h6n5"
server_key: "2600d3430e663c6f67ec06264abd8da9"
server_key_iv: "4ceab0bfdf987f2d1c718608e69d11bf"
server_key_version: "encryptionkey@0"
I20260620 14:23:02.841506 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:02.849681 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:02.851883 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:02.852029 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:02.856155 20552 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42479
I20260620 14:23:02.856186 20619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42479 every 8 connection(s)
I20260620 14:23:02.856525 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:02.857318 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:02.858939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20552
I20260620 14:23:02.859067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:02.859259 392 external_mini_cluster.cc:1428] Setting key 0c2af969244c16454dc62c0c6097a783
I20260620 14:23:02.859941 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0: Bootstrap starting.
I20260620 14:23:02.860484 20620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:02.860805 20620 log.cc:824] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:02.861559 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0: No bootstrap required, opened a new log
I20260620 14:23:02.863725 20620 raft_consensus.cc:348] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27c91913e324b6e8e81801c846ad5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:23:02.863857 20620 raft_consensus.cc:374] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:02.863880 20620 raft_consensus.cc:729] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c27c91913e324b6e8e81801c846ad5c0, State: Initialized, Role: FOLLOWER
I20260620 14:23:02.864013 20620 consensus_queue.cc:260] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27c91913e324b6e8e81801c846ad5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:23:02.864113 20620 raft_consensus.cc:388] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:02.864156 20620 raft_consensus.cc:482] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:02.864202 20620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:02.864893 20620 raft_consensus.cc:504] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27c91913e324b6e8e81801c846ad5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:23:02.865015 20620 leader_election.cc:302] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c27c91913e324b6e8e81801c846ad5c0; no voters:
I20260620 14:23:02.865188 20620 leader_election.cc:288] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:02.865218 20625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:02.865429 20625 raft_consensus.cc:686] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [term 1 LEADER]: Becoming Leader. State: Replica: c27c91913e324b6e8e81801c846ad5c0, State: Running, Role: LEADER
I20260620 14:23:02.865453 20620 sys_catalog.cc:564] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:02.865513 20625 consensus_queue.cc:237] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27c91913e324b6e8e81801c846ad5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:23:02.865811 20626 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c27c91913e324b6e8e81801c846ad5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27c91913e324b6e8e81801c846ad5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260620 14:23:02.865897 20626 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:02.866061 20627 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27c91913e324b6e8e81801c846ad5c0. Latest consensus state: current_term: 1 leader_uuid: "c27c91913e324b6e8e81801c846ad5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27c91913e324b6e8e81801c846ad5c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260620 14:23:02.866127 20627 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:02.866102 20632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 14:23:02 dist-test-slave-h6n5 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:23:02.866673 20632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:02.868445 20632 catalog_manager.cc:1347] Generated new cluster ID: 9066f764b74840aba47bb0aebf72092f
I20260620 14:23:02.868508 20632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:02.871062 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:02.859764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47357 (local address 127.0.0.1:42479)
0620 14:23:02.859919 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:02.859928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:02.859961 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:23:02.860426 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:02.860433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:02.860477 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:02.860630 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:02.861408 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:02.862331 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:02.863461 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:02.864005 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:02.864097 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:02.866975 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:02.866979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:02.866979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:02.866991 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:02.869133 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:02.869594 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:02.869597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:02.869598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:02.869743 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:02.870016 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:02.870017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:02.870018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:02.870305 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:02.870429 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:02.870730 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:23:02.870846 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260620 14:23:02.872236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:02.882274 20632 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:02.883014 20632 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:02.887789 20632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c27c91913e324b6e8e81801c846ad5c0: Generated new TSK 0
I20260620 14:23:02.888010 20632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:02.889227 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:02.882093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47447 (local address 127.0.0.1:42479)
0620 14:23:02.882143 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:02.882145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:02.882281 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:23:02.882405 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:02.882406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:02.882449 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:02.882526 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:02.883166 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:02.883724 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:02.884447 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:02.884669 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:02.884757 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:02.886028 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:02.886029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:02.886030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:02.886037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:02.888145 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:02.888484 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:02.888486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:02.888486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:02.888550 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:02.888749 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:02.888749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:02.888750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:02.888881 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:02.888986 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:02.889072 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:23:02.889135 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:23:02.891315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20552
W20260620 14:23:02.895613 20537 connection.cc:535] client connection to 127.0.0.1:42479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42479): Connection reset by peer (error 104)
2026-06-20T14:23:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:02 dist-test-slave-h6n5 krb5kdc[20651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:02 dist-test-slave-h6n5 krb5kdc[20651](info): set up 2 sockets
Jun 20 14:23:02 dist-test-slave-h6n5 krb5kdc[20651](info): commencing operation
krb5kdc: starting...
W20260620 14:23:03.931527 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:04 dist-test-slave-h6n5 krb5kdc[20651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:04.049808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:04.129463 20664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:04.129709 20664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:04.129750 20664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:04.130916 20664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:04.130980 20664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:04.131002 20664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:04.131024 20664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:04.131050 20664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:04.132524 20664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:04.132790 20664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:04.133011 20664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:04.137794 20672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.137776 20669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:04.137755 20670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:04.137956 20664 server_base.cc:1034] running on GCE node
I20260620 14:23:04.138191 20664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:04.138463 20664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:04.139633 20664 hybrid_clock.cc:648] HybridClock initialized: now 1781965384139611 us; error 27 us; skew 500 ppm
Jun 20 14:23:04 dist-test-slave-h6n5 krb5kdc[20651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:04.146428 20664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:04.146862 20664 webserver.cc:460] Webserver started at http://127.0.0.1:40483/ using document root <none> and password file <none>
I20260620 14:23:04.147094 20664 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:04.147158 20664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:04.147301 20664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:04.148411 20664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "abfec02c87014d49aee124d5694e9825"
format_stamp: "Formatted at 2026-06-20 14:23:04 on dist-test-slave-h6n5"
server_key: "b17f0bd16ae7133015b89357b9d653f5"
server_key_iv: "b1ac8463cc1ded7dd0b0042ce7631b1a"
server_key_version: "encryptionkey@0"
I20260620 14:23:04.148762 20664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "abfec02c87014d49aee124d5694e9825"
format_stamp: "Formatted at 2026-06-20 14:23:04 on dist-test-slave-h6n5"
server_key: "b17f0bd16ae7133015b89357b9d653f5"
server_key_iv: "b1ac8463cc1ded7dd0b0042ce7631b1a"
server_key_version: "encryptionkey@0"
I20260620 14:23:04.150044 20664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:23:04.150756 20679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:04.150957 20664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:04.151060 20664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "abfec02c87014d49aee124d5694e9825"
format_stamp: "Formatted at 2026-06-20 14:23:04 on dist-test-slave-h6n5"
server_key: "b17f0bd16ae7133015b89357b9d653f5"
server_key_iv: "b1ac8463cc1ded7dd0b0042ce7631b1a"
server_key_version: "encryptionkey@0"
I20260620 14:23:04.151139 20664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:04.165309 20664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:04.167309 20664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:04.167481 20664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:04.171327 20664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41371
I20260620 14:23:04.171380 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41371 every 8 connection(s)
I20260620 14:23:04.171789 20664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:04.172652 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:04.174458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20664
I20260620 14:23:04.174669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:04.174916 392 external_mini_cluster.cc:1428] Setting key 9b5521fb40cd391a3f92b97d93fc79df
I20260620 14:23:04.175452 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825: Bootstrap starting.
I20260620 14:23:04.176144 20732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:04.176430 20732 log.cc:824] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:04.177088 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825: No bootstrap required, opened a new log
I20260620 14:23:04.178822 20732 raft_consensus.cc:348] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfec02c87014d49aee124d5694e9825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:23:04.178977 20732 raft_consensus.cc:374] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:04.179020 20732 raft_consensus.cc:729] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfec02c87014d49aee124d5694e9825, State: Initialized, Role: FOLLOWER
I20260620 14:23:04.179158 20732 consensus_queue.cc:260] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfec02c87014d49aee124d5694e9825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:23:04.179248 20732 raft_consensus.cc:388] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:04.179291 20732 raft_consensus.cc:482] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:04.179342 20732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:04.180015 20732 raft_consensus.cc:504] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfec02c87014d49aee124d5694e9825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:23:04.180133 20732 leader_election.cc:302] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abfec02c87014d49aee124d5694e9825; no voters:
I20260620 14:23:04.180315 20732 leader_election.cc:288] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:04.180354 20737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:04.180502 20737 raft_consensus.cc:686] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [term 1 LEADER]: Becoming Leader. State: Replica: abfec02c87014d49aee124d5694e9825, State: Running, Role: LEADER
I20260620 14:23:04.180593 20732 sys_catalog.cc:564] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:04.180665 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.175639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47331 (local address 127.0.0.1:41371)
0620 14:23:04.175821 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.175830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.175862 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:23:04.176341 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.176349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.176399 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.176555 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:04.176796 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.177736 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.178538 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.178948 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.179062 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.179916 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:04.179919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:04.179920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:04.180098 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:04.180156 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:04.180171 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:04.180255 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.180404 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.180437 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260620 14:23:04.180594 20737 consensus_queue.cc:237] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfec02c87014d49aee124d5694e9825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:23:04.180968 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfec02c87014d49aee124d5694e9825. Latest consensus state: current_term: 1 leader_uuid: "abfec02c87014d49aee124d5694e9825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfec02c87014d49aee124d5694e9825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } } }
I20260620 14:23:04.181051 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:04.181214 20738 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abfec02c87014d49aee124d5694e9825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfec02c87014d49aee124d5694e9825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } } }
I20260620 14:23:04.181280 20738 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:04.181252 20741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:04.181610 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:04.181890 20741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:04.183815 20741 catalog_manager.cc:1347] Generated new cluster ID: bc1762dda2a24158924f0cc61130277a
I20260620 14:23:04.183871 20741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:04.186229 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:04.183290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51813 (local address 127.0.0.1:41371)
0620 14:23:04.183329 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:04.183331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:04.183453 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:23:04.183504 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:04.183504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:04.183514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:04.183608 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:04.183743 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.184291 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.185023 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:04.185240 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:04.185327 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:04.185989 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:04.185991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:04.185991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:04.185995 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:04.186009 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:04.186013 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:04.186066 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:04.186124 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:23:04.186133 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:23:04.206548 20741 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:04.207093 20741 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:04.209935 20741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abfec02c87014d49aee124d5694e9825: Generated new TSK 0
I20260620 14:23:04.210137 20741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:04.220073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20664
W20260620 14:23:04.224447 20649 connection.cc:535] client connection to 127.0.0.1:41371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41371): Connection reset by peer (error 104)
2026-06-20T14:23:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:04 dist-test-slave-h6n5 krb5kdc[20763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:04 dist-test-slave-h6n5 krb5kdc[20763](info): set up 2 sockets
Jun 20 14:23:04 dist-test-slave-h6n5 krb5kdc[20763](info): commencing operation
krb5kdc: starting...
W20260620 14:23:05.251657 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:05 dist-test-slave-h6n5 krb5kdc[20763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:05.368579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:05.447715 20776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:05.447917 20776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:05.447942 20776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:05.449113 20776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:05.449157 20776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:05.449172 20776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:05.449184 20776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:05.449196 20776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:05.450702 20776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:05.450963 20776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:05.451195 20776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:05.456025 20781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:05.456033 20784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:05.456032 20782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:05.456719 20776 server_base.cc:1034] running on GCE node
I20260620 14:23:05.456969 20776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:05.457262 20776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:05.458436 20776 hybrid_clock.cc:648] HybridClock initialized: now 1781965385458418 us; error 40 us; skew 500 ppm
Jun 20 14:23:05 dist-test-slave-h6n5 krb5kdc[20763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:05.460923 20776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:05.461298 20776 webserver.cc:460] Webserver started at http://127.0.0.1:33255/ using document root <none> and password file <none>
I20260620 14:23:05.461488 20776 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:05.461525 20776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:05.461615 20776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:05.462638 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "655782a4b2974b24af31879081777bef"
format_stamp: "Formatted at 2026-06-20 14:23:05 on dist-test-slave-h6n5"
server_key: "4aac4863ec0e5b7538b1ba6c89bdd530"
server_key_iv: "2f83dd80483478b6ac2bd6b114d1e1cb"
server_key_version: "encryptionkey@0"
I20260620 14:23:05.462929 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "655782a4b2974b24af31879081777bef"
format_stamp: "Formatted at 2026-06-20 14:23:05 on dist-test-slave-h6n5"
server_key: "4aac4863ec0e5b7538b1ba6c89bdd530"
server_key_iv: "2f83dd80483478b6ac2bd6b114d1e1cb"
server_key_version: "encryptionkey@0"
I20260620 14:23:05.464246 20776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:23:05.464956 20791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:05.465119 20776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:05.465188 20776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "655782a4b2974b24af31879081777bef"
format_stamp: "Formatted at 2026-06-20 14:23:05 on dist-test-slave-h6n5"
server_key: "4aac4863ec0e5b7538b1ba6c89bdd530"
server_key_iv: "2f83dd80483478b6ac2bd6b114d1e1cb"
server_key_version: "encryptionkey@0"
I20260620 14:23:05.465240 20776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:05.468403 20776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:05.470340 20776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:05.470441 20776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:05.474402 20776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44279
I20260620 14:23:05.474470 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44279 every 8 connection(s)
I20260620 14:23:05.474767 20776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:05.475667 20844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:05.477993 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef: Bootstrap starting.
I20260620 14:23:05.478562 20844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:05.478850 20844 log.cc:824] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:05.479487 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef: No bootstrap required, opened a new log
I20260620 14:23:05.481096 20844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655782a4b2974b24af31879081777bef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44279 } }
I20260620 14:23:05.481258 20844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:05.481318 20844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655782a4b2974b24af31879081777bef, State: Initialized, Role: FOLLOWER
I20260620 14:23:05.481494 20844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655782a4b2974b24af31879081777bef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44279 } }
I20260620 14:23:05.481601 20844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:05.481645 20844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:05.481701 20844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:05.482229 20844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655782a4b2974b24af31879081777bef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44279 } }
I20260620 14:23:05.482342 20844 leader_election.cc:302] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 655782a4b2974b24af31879081777bef; no voters:
I20260620 14:23:05.482517 20844 leader_election.cc:288] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:05.482547 20847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:05.482824 20844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:05.482860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20776
I20260620 14:23:05.482872 20847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [term 1 LEADER]: Becoming Leader. State: Replica: 655782a4b2974b24af31879081777bef, State: Running, Role: LEADER
I20260620 14:23:05.482981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:05.483059 20847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655782a4b2974b24af31879081777bef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44279 } }
I20260620 14:23:05.483181 392 external_mini_cluster.cc:1428] Setting key 60866249c624715f129b9046a397ff1a
I20260620 14:23:05.483428 20848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "655782a4b2974b24af31879081777bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655782a4b2974b24af31879081777bef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44279 } } }
I20260620 14:23:05.483520 20848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:05.483544 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 655782a4b2974b24af31879081777bef. Latest consensus state: current_term: 1 leader_uuid: "655782a4b2974b24af31879081777bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655782a4b2974b24af31879081777bef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44279 } } }
I20260620 14:23:05.483705 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:05.483901 20856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:05.484687 20856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:05.486229 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.483853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56825 (local address 127.0.0.1:44279)
0620 14:23:05.484273 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.484283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.484330 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:23:05.484465 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.484473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.484503 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.484642 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:05.485627 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:05.485631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:05.485632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:05.485636 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:05.485685 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:05.485697 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:05.485805 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.485984 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.486027 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":108,"threads_started":1}
I20260620 14:23:05.486346 20856 catalog_manager.cc:1347] Generated new cluster ID: 56ca8495044a4c9c9b4457863ab437e5
I20260620 14:23:05.486405 20856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:05.487102 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:05.492781 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:05.488492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48729 (local address 127.0.0.1:44279)
0620 14:23:05.488533 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:05.488535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:05.491745 (+ 3210us) server_negotiation.cc:407] Connection header received
0620 14:23:05.491843 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:05.491844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:05.491850 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:05.491912 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:05.492526 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:05.492528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:05.492528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:05.492533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:05.492547 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:05.492552 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:05.492601 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:05.492659 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:23:05.492673 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:23:05.508363 20856 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:05.508579 20856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:05.510766 20856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 655782a4b2974b24af31879081777bef: Generated new TSK 0
I20260620 14:23:05.510957 20856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:05.515766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20776
2026-06-20T14:23:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:05 dist-test-slave-h6n5 krb5kdc[20875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:05 dist-test-slave-h6n5 krb5kdc[20875](info): set up 2 sockets
Jun 20 14:23:05 dist-test-slave-h6n5 krb5kdc[20875](info): commencing operation
krb5kdc: starting...
W20260620 14:23:06.551503 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:06 dist-test-slave-h6n5 krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:06.666607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:06.749894 20888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:06.750104 20888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:06.750128 20888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:06.751209 20888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:06.751251 20888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:06.751266 20888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:06.751279 20888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:06.751292 20888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:06.752777 20888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:06.752970 20888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:06.753137 20888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:06.758028 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:06.758023 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:06.758028 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:06.758060 20888 server_base.cc:1034] running on GCE node
I20260620 14:23:06.758515 20888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:06.758803 20888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:06.759995 20888 hybrid_clock.cc:648] HybridClock initialized: now 1781965386759983 us; error 42 us; skew 500 ppm
Jun 20 14:23:06 dist-test-slave-h6n5 krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:06.762621 20888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:06.763028 20888 webserver.cc:460] Webserver started at http://127.0.0.1:44297/ using document root <none> and password file <none>
I20260620 14:23:06.763227 20888 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:06.763275 20888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:06.763365 20888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:06.764441 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "fd70a821e441499890f4a80bf727a29e"
format_stamp: "Formatted at 2026-06-20 14:23:06 on dist-test-slave-h6n5"
server_key: "ef6c64ea4daf10dabdf1fae37ca6fe11"
server_key_iv: "affb6446f56d0067c0eae4866123c63c"
server_key_version: "encryptionkey@0"
I20260620 14:23:06.764746 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd70a821e441499890f4a80bf727a29e"
format_stamp: "Formatted at 2026-06-20 14:23:06 on dist-test-slave-h6n5"
server_key: "ef6c64ea4daf10dabdf1fae37ca6fe11"
server_key_iv: "affb6446f56d0067c0eae4866123c63c"
server_key_version: "encryptionkey@0"
I20260620 14:23:06.766007 20888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:23:06.766796 20903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:06.766970 20888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:06.767055 20888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "fd70a821e441499890f4a80bf727a29e"
format_stamp: "Formatted at 2026-06-20 14:23:06 on dist-test-slave-h6n5"
server_key: "ef6c64ea4daf10dabdf1fae37ca6fe11"
server_key_iv: "affb6446f56d0067c0eae4866123c63c"
server_key_version: "encryptionkey@0"
I20260620 14:23:06.767112 20888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:06.778765 20888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:06.780642 20888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:06.780792 20888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:06.784745 20888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37539
I20260620 14:23:06.784776 20955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37539 every 8 connection(s)
I20260620 14:23:06.785131 20888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:06.785982 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:06.788297 20956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e: Bootstrap starting.
I20260620 14:23:06.788897 20956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:06.789176 20956 log.cc:824] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:06.789880 20956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e: No bootstrap required, opened a new log
I20260620 14:23:06.791047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 20888
I20260620 14:23:06.791175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:06.791347 392 external_mini_cluster.cc:1428] Setting key c5464ec067853af097dbd0c9568cd43b
I20260620 14:23:06.791970 20956 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd70a821e441499890f4a80bf727a29e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260620 14:23:06.792143 20956 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:06.792171 20956 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd70a821e441499890f4a80bf727a29e, State: Initialized, Role: FOLLOWER
I20260620 14:23:06.792323 20956 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd70a821e441499890f4a80bf727a29e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260620 14:23:06.792397 20956 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:06.792418 20956 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:06.792481 20956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:06.793306 20956 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd70a821e441499890f4a80bf727a29e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260620 14:23:06.793430 20956 leader_election.cc:302] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd70a821e441499890f4a80bf727a29e; no voters:
I20260620 14:23:06.793617 20956 leader_election.cc:288] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:06.793653 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:06.793752 20961 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [term 1 LEADER]: Becoming Leader. State: Replica: fd70a821e441499890f4a80bf727a29e, State: Running, Role: LEADER
I20260620 14:23:06.793877 20956 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:06.793895 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd70a821e441499890f4a80bf727a29e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260620 14:23:06.794319 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd70a821e441499890f4a80bf727a29e. Latest consensus state: current_term: 1 leader_uuid: "fd70a821e441499890f4a80bf727a29e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd70a821e441499890f4a80bf727a29e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260620 14:23:06.794560 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:06.794485 20962 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd70a821e441499890f4a80bf727a29e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd70a821e441499890f4a80bf727a29e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260620 14:23:06.794972 20962 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:06.795197 20970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:06.795857 20970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:06.798477 20970 catalog_manager.cc:1347] Generated new cluster ID: b703d022eed940e9b53d2e5a067dfbf8
I20260620 14:23:06.798533 20970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:23:06 dist-test-slave-h6n5 krb5kdc[20875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:23:06.803253 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.791931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40775 (local address 127.0.0.1:37539)
0620 14:23:06.792133 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:06.792143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:06.792179 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:23:06.792336 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:06.792343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:06.792386 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:06.792537 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:06.793195 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:06.794452 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:06.795652 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:06.796224 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:06.796647 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:06.799318 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:06.799322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:06.799322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:06.799334 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:06.801558 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:06.801971 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:06.801974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:06.801974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:06.802092 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:06.802303 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:06.802304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:06.802305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:06.802665 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:06.802838 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:06.802905 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:23:06.803047 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260620 14:23:06.804335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:06.812728 20970 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:06.813539 20970 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:06.815068 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:06.805683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49977 (local address 127.0.0.1:37539)
0620 14:23:06.805722 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:06.805724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:06.807726 (+ 2002us) server_negotiation.cc:407] Connection header received
0620 14:23:06.807780 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:06.807780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:06.807790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:06.807842 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:06.808484 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:06.809199 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:06.810221 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:06.810529 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:06.810593 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:06.812196 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:06.812198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:06.812198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:06.812206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:06.813945 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:06.814313 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:06.814314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:06.814314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:06.814369 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:06.814560 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:06.814561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:06.814562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:06.814674 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:06.814780 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:06.814880 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:23:06.814967 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:23:06.818022 20970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd70a821e441499890f4a80bf727a29e: Generated new TSK 0
I20260620 14:23:06.818219 20970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:06.822211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 20888
2026-06-20T14:23:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:06 dist-test-slave-h6n5 krb5kdc[20987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:06 dist-test-slave-h6n5 krb5kdc[20987](info): set up 2 sockets
Jun 20 14:23:06 dist-test-slave-h6n5 krb5kdc[20987](info): commencing operation
krb5kdc: starting...
W20260620 14:23:07.863718 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:07 dist-test-slave-h6n5 krb5kdc[20987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:07.989719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:08.069202 21000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:08.069394 21000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:08.069418 21000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:08.070537 21000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:08.070580 21000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:08.070593 21000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:08.070606 21000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:08.070655 21000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:08.072109 21000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:08.072337 21000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:08.072564 21000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:08.077126 21005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:08.077131 21006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:08.077196 21000 server_base.cc:1034] running on GCE node
W20260620 14:23:08.077135 21008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:08.077579 21000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:08.077859 21000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:08.079028 21000 hybrid_clock.cc:648] HybridClock initialized: now 1781965388079013 us; error 27 us; skew 500 ppm
Jun 20 14:23:08 dist-test-slave-h6n5 krb5kdc[20987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:08.081521 21000 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:08.081910 21000 webserver.cc:460] Webserver started at http://127.0.0.1:45949/ using document root <none> and password file <none>
I20260620 14:23:08.082132 21000 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:08.082195 21000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:08.082353 21000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:08.083355 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "e697f1f3c1474a1594b7625c06658903"
format_stamp: "Formatted at 2026-06-20 14:23:08 on dist-test-slave-h6n5"
server_key: "6c32bc2a99b94de7d593e5228299eeab"
server_key_iv: "18b49f3e01ad2dbb8280e1d62718fa86"
server_key_version: "encryptionkey@0"
I20260620 14:23:08.083724 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "e697f1f3c1474a1594b7625c06658903"
format_stamp: "Formatted at 2026-06-20 14:23:08 on dist-test-slave-h6n5"
server_key: "6c32bc2a99b94de7d593e5228299eeab"
server_key_iv: "18b49f3e01ad2dbb8280e1d62718fa86"
server_key_version: "encryptionkey@0"
I20260620 14:23:08.084990 21000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:08.085670 21015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:08.085822 21000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:08.085916 21000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "e697f1f3c1474a1594b7625c06658903"
format_stamp: "Formatted at 2026-06-20 14:23:08 on dist-test-slave-h6n5"
server_key: "6c32bc2a99b94de7d593e5228299eeab"
server_key_iv: "18b49f3e01ad2dbb8280e1d62718fa86"
server_key_version: "encryptionkey@0"
I20260620 14:23:08.085991 21000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:08.114354 21000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:08.116322 21000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:08.116490 21000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:08.120375 21000 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40007
I20260620 14:23:08.120415 21067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40007 every 8 connection(s)
I20260620 14:23:08.120738 21000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:08.121556 21068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:08.123867 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903: Bootstrap starting.
I20260620 14:23:08.124485 21068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:08.124511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21000
I20260620 14:23:08.124631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:08.124850 392 external_mini_cluster.cc:1428] Setting key 46189600b39367cdffb9cf08a8b3c481
I20260620 14:23:08.124847 21068 log.cc:824] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:08.125627 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903: No bootstrap required, opened a new log
I20260620 14:23:08.128322 21068 raft_consensus.cc:348] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e697f1f3c1474a1594b7625c06658903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260620 14:23:08.128456 21068 raft_consensus.cc:374] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:08.128479 21068 raft_consensus.cc:729] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e697f1f3c1474a1594b7625c06658903, State: Initialized, Role: FOLLOWER
I20260620 14:23:08.128605 21068 consensus_queue.cc:260] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e697f1f3c1474a1594b7625c06658903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260620 14:23:08.128675 21068 raft_consensus.cc:388] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:08.128715 21068 raft_consensus.cc:482] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:08.128762 21068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:08.129427 21068 raft_consensus.cc:504] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e697f1f3c1474a1594b7625c06658903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260620 14:23:08.129531 21068 leader_election.cc:302] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e697f1f3c1474a1594b7625c06658903; no voters:
I20260620 14:23:08.129686 21068 leader_election.cc:288] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:08.129746 21073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:08.129948 21073 raft_consensus.cc:686] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [term 1 LEADER]: Becoming Leader. State: Replica: e697f1f3c1474a1594b7625c06658903, State: Running, Role: LEADER
I20260620 14:23:08.130015 21068 sys_catalog.cc:564] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:08.130448 21073 consensus_queue.cc:237] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e697f1f3c1474a1594b7625c06658903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } }
I20260620 14:23:08.130848 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e697f1f3c1474a1594b7625c06658903. Latest consensus state: current_term: 1 leader_uuid: "e697f1f3c1474a1594b7625c06658903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e697f1f3c1474a1594b7625c06658903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } } }
I20260620 14:23:08.131161 21074 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e697f1f3c1474a1594b7625c06658903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e697f1f3c1474a1594b7625c06658903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40007 } } }
I20260620 14:23:08.131467 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:08.131461 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:08.125316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55907 (local address 127.0.0.1:40007)
0620 14:23:08.125503 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:08.125512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:08.125543 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:23:08.126085 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:08.126094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:08.126157 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:08.126305 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:08.126557 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:08.127999 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:08.129004 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:08.129643 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:08.129765 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:08.130524 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:08.130528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:08.130528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:08.130734 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:08.130772 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:08.130789 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:08.130874 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:08.131183 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:23:08.131246 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260620 14:23:08.131583 21074 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903 [sys.catalog]: This master's current role is: LEADER
W20260620 14:23:08.132186 21088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:08.132373 21088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:08.132498 21089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:08.132537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:08.132673 21089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:08.134241 21089 catalog_manager.cc:1347] Generated new cluster ID: d179d607b577437ebb4b00330cff01f7
I20260620 14:23:08.134292 21089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:08.136963 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:08.134133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58733 (local address 127.0.0.1:40007)
0620 14:23:08.134165 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:08.134167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:08.134276 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:23:08.134378 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:08.134379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:08.134389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:08.134447 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:08.134577 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:08.135055 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:08.135818 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:08.136023 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:08.136092 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:08.136712 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:08.136713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:08.136714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:08.136718 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:08.136731 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:08.136735 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:08.136787 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:08.136849 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:23:08.136867 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:23:08.163106 21089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:08.163741 21089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:08.167125 21089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e697f1f3c1474a1594b7625c06658903: Generated new TSK 0
I20260620 14:23:08.167294 21089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:08.173128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21000
2026-06-20T14:23:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:08 dist-test-slave-h6n5 krb5kdc[21099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:08 dist-test-slave-h6n5 krb5kdc[21099](info): set up 2 sockets
Jun 20 14:23:08 dist-test-slave-h6n5 krb5kdc[21099](info): commencing operation
krb5kdc: starting...
W20260620 14:23:09.219610 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:09 dist-test-slave-h6n5 krb5kdc[21099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:09.336702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:09.417405 21112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:09.417598 21112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:09.417621 21112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:09.418804 21112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:09.418848 21112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:09.418862 21112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:09.418874 21112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:09.418924 21112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:09.420372 21112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:09.420611 21112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:09.420846 21112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:09.425643 21118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:09.425721 21117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:09.425643 21120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:09.425917 21112 server_base.cc:1034] running on GCE node
I20260620 14:23:09.426188 21112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:09.426491 21112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:09.427649 21112 hybrid_clock.cc:648] HybridClock initialized: now 1781965389427632 us; error 38 us; skew 500 ppm
Jun 20 14:23:09 dist-test-slave-h6n5 krb5kdc[21099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:09.436199 21112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:09.436627 21112 webserver.cc:460] Webserver started at http://127.0.0.1:44185/ using document root <none> and password file <none>
I20260620 14:23:09.436807 21112 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:09.436847 21112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:09.436935 21112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:09.437950 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "74cc1f88b026426b8b8f7d2e57ab38c5"
format_stamp: "Formatted at 2026-06-20 14:23:09 on dist-test-slave-h6n5"
server_key: "5fd183ad6cd8e210ab8c0bfd60084a7c"
server_key_iv: "888edfec25469f373980931527661e94"
server_key_version: "encryptionkey@0"
I20260620 14:23:09.438249 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "74cc1f88b026426b8b8f7d2e57ab38c5"
format_stamp: "Formatted at 2026-06-20 14:23:09 on dist-test-slave-h6n5"
server_key: "5fd183ad6cd8e210ab8c0bfd60084a7c"
server_key_iv: "888edfec25469f373980931527661e94"
server_key_version: "encryptionkey@0"
I20260620 14:23:09.439495 21112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:23:09.440332 21127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:09.440515 21112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:09.440603 21112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "74cc1f88b026426b8b8f7d2e57ab38c5"
format_stamp: "Formatted at 2026-06-20 14:23:09 on dist-test-slave-h6n5"
server_key: "5fd183ad6cd8e210ab8c0bfd60084a7c"
server_key_iv: "888edfec25469f373980931527661e94"
server_key_version: "encryptionkey@0"
I20260620 14:23:09.440657 21112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:09.443938 21112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:09.445824 21112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:09.445935 21112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:09.449903 21112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46679
I20260620 14:23:09.449978 21179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46679 every 8 connection(s)
I20260620 14:23:09.450274 21112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:09.451318 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:09.451675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21112
I20260620 14:23:09.451792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:09.452005 392 external_mini_cluster.cc:1428] Setting key 75fba98746f2c83a81a621d74a226056
I20260620 14:23:09.454645 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:09.452525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44297 (local address 127.0.0.1:46679)
0620 14:23:09.452790 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:09.452800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:09.452832 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:23:09.453308 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:09.453317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:09.453362 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:09.453514 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:09.454175 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:09.454179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:09.454180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:09.454184 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:09.454228 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:09.454242 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:09.454321 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:09.454414 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:23:09.454454 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":62,"threads_started":1}
I20260620 14:23:09.455140 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5: Bootstrap starting.
I20260620 14:23:09.455863 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:09.455914 21180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:09.456188 21180 log.cc:824] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:09.456972 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5: No bootstrap required, opened a new log
I20260620 14:23:09.458488 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:09.457465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34665 (local address 127.0.0.1:46679)
0620 14:23:09.457503 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:09.457504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:09.457612 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:23:09.457665 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:09.457666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:09.457671 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:09.457725 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:09.458281 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:09.458283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:09.458283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:09.458287 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:09.458302 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:09.458306 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:09.458376 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:09.458411 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:23:09.458424 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:23:09.459532 21180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260620 14:23:09.459743 21180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:09.459767 21180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cc1f88b026426b8b8f7d2e57ab38c5, State: Initialized, Role: FOLLOWER
I20260620 14:23:09.459890 21180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260620 14:23:09.459975 21180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:09.459997 21180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:09.460026 21180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:09.460522 21180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260620 14:23:09.460608 21180 leader_election.cc:302] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74cc1f88b026426b8b8f7d2e57ab38c5; no voters:
I20260620 14:23:09.460731 21180 leader_election.cc:288] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:09.460795 21190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:09.460956 21190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [term 1 LEADER]: Becoming Leader. State: Replica: 74cc1f88b026426b8b8f7d2e57ab38c5, State: Running, Role: LEADER
I20260620 14:23:09.460994 21180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:09.461072 21190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260620 14:23:09.461454 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } } }
I20260620 14:23:09.461457 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74cc1f88b026426b8b8f7d2e57ab38c5. Latest consensus state: current_term: 1 leader_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cc1f88b026426b8b8f7d2e57ab38c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } } }
I20260620 14:23:09.461599 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:09.461614 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5 [sys.catalog]: This master's current role is: LEADER
W20260620 14:23:09.462725 21206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:09.462769 21206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:09.462836 21199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:09.462983 21199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:09.464385 21199 catalog_manager.cc:1347] Generated new cluster ID: b43fdba6bd644479b06d3f98cfadc65a
I20260620 14:23:09.464473 21199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:09.471616 21199 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:09.471779 21199 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:09.481353 21199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74cc1f88b026426b8b8f7d2e57ab38c5: Generated new TSK 0
I20260620 14:23:09.481557 21199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:09.484416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21112
2026-06-20T14:23:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:09 dist-test-slave-h6n5 krb5kdc[21211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:09 dist-test-slave-h6n5 krb5kdc[21211](info): set up 2 sockets
Jun 20 14:23:09 dist-test-slave-h6n5 krb5kdc[21211](info): commencing operation
krb5kdc: starting...
W20260620 14:23:10.519722 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:10 dist-test-slave-h6n5 krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:10.638275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:10.719127 21224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:10.719311 21224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:10.719334 21224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:10.721151 21224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:10.721194 21224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:10.721208 21224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:10.721221 21224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:10.721258 21224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:10.722811 21224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:10.723054 21224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:10.723292 21224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:10.727979 21229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:10.727975 21230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:10.727972 21232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:10.728505 21224 server_base.cc:1034] running on GCE node
I20260620 14:23:10.728699 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:10.728945 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:10.730084 21224 hybrid_clock.cc:648] HybridClock initialized: now 1781965390730071 us; error 29 us; skew 500 ppm
Jun 20 14:23:10 dist-test-slave-h6n5 krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:10.732471 21224 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:10.732913 21224 webserver.cc:460] Webserver started at http://127.0.0.1:35255/ using document root <none> and password file <none>
I20260620 14:23:10.733141 21224 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:10.733186 21224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:10.733351 21224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:10.734373 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "45f0cc190565486d8d495d0463df06b8"
format_stamp: "Formatted at 2026-06-20 14:23:10 on dist-test-slave-h6n5"
server_key: "661562cbf3203f25a4016ed2fb2e10b2"
server_key_iv: "b1ceca85881a7dd4c73207c246808667"
server_key_version: "encryptionkey@0"
I20260620 14:23:10.734711 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "45f0cc190565486d8d495d0463df06b8"
format_stamp: "Formatted at 2026-06-20 14:23:10 on dist-test-slave-h6n5"
server_key: "661562cbf3203f25a4016ed2fb2e10b2"
server_key_iv: "b1ceca85881a7dd4c73207c246808667"
server_key_version: "encryptionkey@0"
I20260620 14:23:10.736045 21224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:10.736768 21239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:10.736955 21224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:10.737068 21224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "45f0cc190565486d8d495d0463df06b8"
format_stamp: "Formatted at 2026-06-20 14:23:10 on dist-test-slave-h6n5"
server_key: "661562cbf3203f25a4016ed2fb2e10b2"
server_key_iv: "b1ceca85881a7dd4c73207c246808667"
server_key_version: "encryptionkey@0"
I20260620 14:23:10.737142 21224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:10.753778 21224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:10.755575 21224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:10.755731 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:10.759408 21224 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34265
I20260620 14:23:10.759482 21291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34265 every 8 connection(s)
I20260620 14:23:10.759835 21224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:10.760691 21292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:10.762887 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8: Bootstrap starting.
I20260620 14:23:10.763183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21224
I20260620 14:23:10.763322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:10.763494 21292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:10.763513 392 external_mini_cluster.cc:1428] Setting key 4c3f48e1d90a150f8e2b44f8d1043a98
I20260620 14:23:10.763808 21292 log.cc:824] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:10.764750 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8: No bootstrap required, opened a new log
I20260620 14:23:10.767062 21292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f0cc190565486d8d495d0463df06b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260620 14:23:10.767236 21292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:10.767272 21292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0cc190565486d8d495d0463df06b8, State: Initialized, Role: FOLLOWER
I20260620 14:23:10.767403 21292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f0cc190565486d8d495d0463df06b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260620 14:23:10.767486 21292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:10.767530 21292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:10.767599 21292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:10.768178 21292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f0cc190565486d8d495d0463df06b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260620 14:23:10.768280 21292 leader_election.cc:302] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f0cc190565486d8d495d0463df06b8; no voters:
I20260620 14:23:10.768436 21292 leader_election.cc:288] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:10.768509 21297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:10.768685 21292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:10.768693 21297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [term 1 LEADER]: Becoming Leader. State: Replica: 45f0cc190565486d8d495d0463df06b8, State: Running, Role: LEADER
I20260620 14:23:10.768873 21297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f0cc190565486d8d495d0463df06b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260620 14:23:10.769234 21298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f0cc190565486d8d495d0463df06b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f0cc190565486d8d495d0463df06b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260620 14:23:10.769381 21298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:10.769505 21299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f0cc190565486d8d495d0463df06b8. Latest consensus state: current_term: 1 leader_uuid: "45f0cc190565486d8d495d0463df06b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f0cc190565486d8d495d0463df06b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260620 14:23:10.769603 21299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8 [sys.catalog]: This master's current role is: LEADER
W20260620 14:23:10.770802 21312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:10.770871 21312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:10.770982 21306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:10.771169 21306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:23:10 dist-test-slave-h6n5 krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:23:10.772794 21306 catalog_manager.cc:1347] Generated new cluster ID: e1a4d55bdba24b2aba17d41a7c91bfcd
I20260620 14:23:10.772874 21306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:10.775825 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:10.764026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:39493 (local address 127.0.0.1:34265)
0620 14:23:10.764204 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:10.764215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:10.764247 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:23:10.764822 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:10.764831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:10.764905 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:10.765058 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:10.765655 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:10.767117 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:10.768101 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:10.768722 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:10.769185 (+ 463us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:10.771931 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:10.771935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:10.771936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:10.771947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:10.774058 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:10.774506 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:10.774509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:10.774510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:10.774625 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:10.774859 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:10.774860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:10.774860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:10.775168 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:10.775341 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:10.775448 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:23:10.775592 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260620 14:23:10.776909 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:10.785600 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:10.778298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:56301 (local address 127.0.0.1:34265)
0620 14:23:10.778337 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:10.778338 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:10.778426 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:23:10.778457 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:10.778458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:10.778468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:10.778533 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:10.779127 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:10.779914 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:10.780872 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:10.781205 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:10.781283 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:10.782675 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:10.782677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:10.782677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:10.782684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:10.784352 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:10.784776 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:10.784777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:10.784777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:10.784839 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:10.785058 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:10.785059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:10.785060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:10.785212 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:10.785326 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:10.785436 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:23:10.785502 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:23:10.794377 21306 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:10.795136 21306 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:10.799604 21306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f0cc190565486d8d495d0463df06b8: Generated new TSK 0
I20260620 14:23:10.799817 21306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:10.804754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21224
W20260620 14:23:10.808979 21209 connection.cc:535] client connection to 127.0.0.1:34265 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34265): Connection reset by peer (error 104)
2026-06-20T14:23:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:10 dist-test-slave-h6n5 krb5kdc[21323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:10 dist-test-slave-h6n5 krb5kdc[21323](info): set up 2 sockets
Jun 20 14:23:10 dist-test-slave-h6n5 krb5kdc[21323](info): commencing operation
krb5kdc: starting...
W20260620 14:23:11.839681 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:11 dist-test-slave-h6n5 krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:11.960659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:12.040339 21336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:12.040545 21336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:12.040568 21336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:12.041785 21336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:12.041828 21336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:12.041842 21336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:12.041860 21336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:12.041872 21336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:12.043303 21336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:12.043498 21336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:12.043740 21336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:23:12.048691 21336 server_base.cc:1034] running on GCE node
W20260620 14:23:12.048653 21342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:12.048660 21344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:12.048660 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:12.049180 21336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:12.049434 21336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:12.050590 21336 hybrid_clock.cc:648] HybridClock initialized: now 1781965392050575 us; error 36 us; skew 500 ppm
Jun 20 14:23:12 dist-test-slave-h6n5 krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:12.053181 21336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:12.053581 21336 webserver.cc:460] Webserver started at http://127.0.0.1:34849/ using document root <none> and password file <none>
I20260620 14:23:12.053771 21336 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:12.053812 21336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:12.053902 21336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:12.054924 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "2018b4c775bd4c9aaef6997e333a6500"
format_stamp: "Formatted at 2026-06-20 14:23:12 on dist-test-slave-h6n5"
server_key: "0efd8b621a2bfdf7c430f9d7cc906b21"
server_key_iv: "52f502a58ff4dc1d9588a3e132af1cda"
server_key_version: "encryptionkey@0"
I20260620 14:23:12.055212 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "2018b4c775bd4c9aaef6997e333a6500"
format_stamp: "Formatted at 2026-06-20 14:23:12 on dist-test-slave-h6n5"
server_key: "0efd8b621a2bfdf7c430f9d7cc906b21"
server_key_iv: "52f502a58ff4dc1d9588a3e132af1cda"
server_key_version: "encryptionkey@0"
I20260620 14:23:12.056535 21336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:12.057212 21351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:12.057411 21336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:12.057524 21336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "2018b4c775bd4c9aaef6997e333a6500"
format_stamp: "Formatted at 2026-06-20 14:23:12 on dist-test-slave-h6n5"
server_key: "0efd8b621a2bfdf7c430f9d7cc906b21"
server_key_iv: "52f502a58ff4dc1d9588a3e132af1cda"
server_key_version: "encryptionkey@0"
I20260620 14:23:12.057605 21336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:12.082233 21336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:12.084693 21336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:12.084863 21336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:12.088886 21336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37295
I20260620 14:23:12.088939 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37295 every 8 connection(s)
I20260620 14:23:12.089308 21336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:12.090178 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:12.092519 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500: Bootstrap starting.
I20260620 14:23:12.093097 21404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:12.093381 21404 log.cc:824] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:12.094059 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500: No bootstrap required, opened a new log
I20260620 14:23:12.095182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21336
I20260620 14:23:12.095301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:12.095491 392 external_mini_cluster.cc:1428] Setting key 24d7a1483001d7ddee1ad3fde6ba410b
I20260620 14:23:12.096073 21404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2018b4c775bd4c9aaef6997e333a6500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 14:23:12.096256 21404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:12.096316 21404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2018b4c775bd4c9aaef6997e333a6500, State: Initialized, Role: FOLLOWER
I20260620 14:23:12.096458 21404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2018b4c775bd4c9aaef6997e333a6500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 14:23:12.096546 21404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:12.096594 21404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:12.096643 21404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:12.097599 21404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2018b4c775bd4c9aaef6997e333a6500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 14:23:12.097685 21404 leader_election.cc:302] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2018b4c775bd4c9aaef6997e333a6500; no voters:
I20260620 14:23:12.097877 21404 leader_election.cc:288] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:12.097898 21409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:12.098013 21409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [term 1 LEADER]: Becoming Leader. State: Replica: 2018b4c775bd4c9aaef6997e333a6500, State: Running, Role: LEADER
I20260620 14:23:12.098140 21404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:12.098152 21409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2018b4c775bd4c9aaef6997e333a6500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 14:23:12.098521 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2018b4c775bd4c9aaef6997e333a6500. Latest consensus state: current_term: 1 leader_uuid: "2018b4c775bd4c9aaef6997e333a6500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2018b4c775bd4c9aaef6997e333a6500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } } }
I20260620 14:23:12.098666 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:12.099054 21418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:12.099375 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2018b4c775bd4c9aaef6997e333a6500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2018b4c775bd4c9aaef6997e333a6500" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } } }
I20260620 14:23:12.099522 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:12.099649 21418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:12.101486 21418 catalog_manager.cc:1347] Generated new cluster ID: 32112367d4b244c29feacc03e6298d75
I20260620 14:23:12.101541 21418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:12.101565 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:12.096053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:59947 (local address 127.0.0.1:37295)
0620 14:23:12.096246 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:12.096256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:12.096290 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:23:12.096412 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:12.096420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:12.096485 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:12.096634 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:12.097006 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:12.098052 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:12.099215 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:12.099874 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:12.100390 (+ 516us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:12.100833 (+ 443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:12.100837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:12.100837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:12.101047 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:12.101102 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:12.101118 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:12.101205 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:12.101319 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:23:12.101347 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260620 14:23:12.102381 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:12.106547 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:12.103681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:56193 (local address 127.0.0.1:37295)
0620 14:23:12.103718 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:12.103721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:12.103872 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:23:12.103935 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:12.103936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:12.103946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:12.104002 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:12.104134 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:12.104608 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:12.105308 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:12.105541 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:12.105624 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:12.106309 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:12.106310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:12.106311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:12.106315 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:12.106328 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:12.106332 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:12.106385 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:12.106463 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:23:12.106474 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:23:12.121905 21418 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:12.122526 21418 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:12.124432 21418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2018b4c775bd4c9aaef6997e333a6500: Generated new TSK 0
I20260620 14:23:12.124614 21418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:12.132054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21336
W20260620 14:23:12.136263 21321 connection.cc:535] client connection to 127.0.0.1:37295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37295): Connection reset by peer (error 104)
2026-06-20T14:23:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:12 dist-test-slave-h6n5 krb5kdc[21435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:12 dist-test-slave-h6n5 krb5kdc[21435](info): set up 2 sockets
Jun 20 14:23:12 dist-test-slave-h6n5 krb5kdc[21435](info): commencing operation
krb5kdc: starting...
W20260620 14:23:13.167794 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:13 dist-test-slave-h6n5 krb5kdc[21435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:13.286073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:13.366904 21448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:13.367141 21448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:13.367179 21448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:13.368494 21448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:13.368564 21448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:13.368589 21448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:13.368620 21448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:13.368647 21448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:13.370108 21448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:13.370378 21448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:13.370597 21448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:13.375523 21454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:13.375514 21456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:13.375491 21453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:13.375655 21448 server_base.cc:1034] running on GCE node
I20260620 14:23:13.375975 21448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:13.376258 21448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:13.377424 21448 hybrid_clock.cc:648] HybridClock initialized: now 1781965393377435 us; error 78 us; skew 500 ppm
Jun 20 14:23:13 dist-test-slave-h6n5 krb5kdc[21435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:13.379808 21448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:13.380250 21448 webserver.cc:460] Webserver started at http://127.0.0.1:34951/ using document root <none> and password file <none>
I20260620 14:23:13.380473 21448 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:13.380542 21448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:13.380687 21448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:13.381697 21448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "4196ac055a6446ad8c384cffa536d838"
format_stamp: "Formatted at 2026-06-20 14:23:13 on dist-test-slave-h6n5"
server_key: "15389ab915ec301990517084cb6b95f7"
server_key_iv: "2d603e7d10bef551c5b654eaa83e2430"
server_key_version: "encryptionkey@0"
I20260620 14:23:13.382047 21448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "4196ac055a6446ad8c384cffa536d838"
format_stamp: "Formatted at 2026-06-20 14:23:13 on dist-test-slave-h6n5"
server_key: "15389ab915ec301990517084cb6b95f7"
server_key_iv: "2d603e7d10bef551c5b654eaa83e2430"
server_key_version: "encryptionkey@0"
I20260620 14:23:13.383425 21448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:23:13.384302 21463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:13.384471 21448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:13.384570 21448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "4196ac055a6446ad8c384cffa536d838"
format_stamp: "Formatted at 2026-06-20 14:23:13 on dist-test-slave-h6n5"
server_key: "15389ab915ec301990517084cb6b95f7"
server_key_iv: "2d603e7d10bef551c5b654eaa83e2430"
server_key_version: "encryptionkey@0"
I20260620 14:23:13.384652 21448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:13.387890 21448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:13.389657 21448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:13.389822 21448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:13.393776 21448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35415
I20260620 14:23:13.393806 21515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35415 every 8 connection(s)
I20260620 14:23:13.394133 21448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:13.395097 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:13.397254 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838: Bootstrap starting.
I20260620 14:23:13.397763 21516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:13.397989 21516 log.cc:824] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:13.398625 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838: No bootstrap required, opened a new log
I20260620 14:23:13.400211 21516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4196ac055a6446ad8c384cffa536d838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35415 } }
I20260620 14:23:13.400339 21516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:13.400362 21516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4196ac055a6446ad8c384cffa536d838, State: Initialized, Role: FOLLOWER
I20260620 14:23:13.400488 21516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4196ac055a6446ad8c384cffa536d838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35415 } }
I20260620 14:23:13.400575 21516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:13.400599 21516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:13.400660 21516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:13.400662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21448
I20260620 14:23:13.400827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:13.401005 392 external_mini_cluster.cc:1428] Setting key 3f12b0933fc61a33ba7b5aaee141bfdd
I20260620 14:23:13.401280 21516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4196ac055a6446ad8c384cffa536d838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35415 } }
I20260620 14:23:13.401396 21516 leader_election.cc:302] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4196ac055a6446ad8c384cffa536d838; no voters:
I20260620 14:23:13.401604 21516 leader_election.cc:288] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:13.401640 21521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:13.401844 21521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [term 1 LEADER]: Becoming Leader. State: Replica: 4196ac055a6446ad8c384cffa536d838, State: Running, Role: LEADER
I20260620 14:23:13.401968 21516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:13.401979 21521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4196ac055a6446ad8c384cffa536d838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35415 } }
I20260620 14:23:13.402693 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4196ac055a6446ad8c384cffa536d838. Latest consensus state: current_term: 1 leader_uuid: "4196ac055a6446ad8c384cffa536d838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4196ac055a6446ad8c384cffa536d838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35415 } } }
I20260620 14:23:13.402709 21522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4196ac055a6446ad8c384cffa536d838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4196ac055a6446ad8c384cffa536d838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35415 } } }
I20260620 14:23:13.402855 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:13.402875 21522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:13.403095 21529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:13.403438 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:13.401442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:58963 (local address 127.0.0.1:35415)
0620 14:23:13.401620 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:13.401630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:13.401727 (+ 97us) server_negotiation.cc:407] Connection header received
0620 14:23:13.401881 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:13.401889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:13.401919 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:13.402389 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:13.402823 (+ 434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:13.402827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:13.402828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:13.402832 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:13.402875 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:13.402890 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:13.403104 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:13.403139 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:23:13.403183 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260620 14:23:13.403913 21529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:13.404386 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:13.406810 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:13.405863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:54549 (local address 127.0.0.1:35415)
0620 14:23:13.405911 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:13.405912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:13.405981 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:23:13.406037 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:13.406037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:13.406041 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:13.406087 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:13.406584 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:13.406585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:13.406586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:13.406589 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:13.406600 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:13.406603 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:13.406646 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:13.406699 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:23:13.406708 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:23:13.408680 21529 catalog_manager.cc:1347] Generated new cluster ID: d010375e3c6341fca2b37a8b0a5fbb19
I20260620 14:23:13.408751 21529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:13.416131 21529 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:13.416273 21529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:13.420040 21529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4196ac055a6446ad8c384cffa536d838: Generated new TSK 0
I20260620 14:23:13.420219 21529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:13.427709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21448
2026-06-20T14:23:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:13 dist-test-slave-h6n5 krb5kdc[21547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:13 dist-test-slave-h6n5 krb5kdc[21547](info): set up 2 sockets
Jun 20 14:23:13 dist-test-slave-h6n5 krb5kdc[21547](info): commencing operation
krb5kdc: starting...
W20260620 14:23:14.472002 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:14 dist-test-slave-h6n5 krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:14.586818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:14.666905 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:14.667140 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:14.667177 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:14.668388 21560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:14.668455 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:14.668479 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:14.668500 21560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:14.668527 21560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:14.669968 21560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:14.670243 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:14.670464 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:14.675161 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:14.675172 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:14.675334 21560 server_base.cc:1034] running on GCE node
W20260620 14:23:14.675380 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:14.675680 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:14.675973 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:14.677150 21560 hybrid_clock.cc:648] HybridClock initialized: now 1781965394677129 us; error 38 us; skew 500 ppm
Jun 20 14:23:14 dist-test-slave-h6n5 krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:14.684711 21560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:14.685194 21560 webserver.cc:460] Webserver started at http://127.0.0.1:37909/ using document root <none> and password file <none>
I20260620 14:23:14.685415 21560 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:14.685479 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:14.685631 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:14.686714 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "5f29d09b02e1449caa04f22caae4ee31"
format_stamp: "Formatted at 2026-06-20 14:23:14 on dist-test-slave-h6n5"
server_key: "eac22c727e8c81f10fee7622eb76f8a1"
server_key_iv: "ec9a6c44d1b8aa068549a70b22cacda0"
server_key_version: "encryptionkey@0"
I20260620 14:23:14.687081 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f29d09b02e1449caa04f22caae4ee31"
format_stamp: "Formatted at 2026-06-20 14:23:14 on dist-test-slave-h6n5"
server_key: "eac22c727e8c81f10fee7622eb76f8a1"
server_key_iv: "ec9a6c44d1b8aa068549a70b22cacda0"
server_key_version: "encryptionkey@0"
I20260620 14:23:14.688499 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:14.689280 21575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:14.689481 21560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:14.689584 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "5f29d09b02e1449caa04f22caae4ee31"
format_stamp: "Formatted at 2026-06-20 14:23:14 on dist-test-slave-h6n5"
server_key: "eac22c727e8c81f10fee7622eb76f8a1"
server_key_iv: "ec9a6c44d1b8aa068549a70b22cacda0"
server_key_version: "encryptionkey@0"
I20260620 14:23:14.689640 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:14.701267 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:14.703128 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:14.703238 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:14.707098 21560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41069
I20260620 14:23:14.707142 21627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41069 every 8 connection(s)
I20260620 14:23:14.707510 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:14.708405 21628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:14.710705 21628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31: Bootstrap starting.
I20260620 14:23:14.711287 21628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:14.711474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21560
I20260620 14:23:14.711621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:14.711668 21628 log.cc:824] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:14.711820 392 external_mini_cluster.cc:1428] Setting key c0e8065854a6abdb25c45c08c15cd28b
I20260620 14:23:14.712566 21628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31: No bootstrap required, opened a new log
I20260620 14:23:14.715250 21628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29d09b02e1449caa04f22caae4ee31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260620 14:23:14.715422 21628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:14.715456 21628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f29d09b02e1449caa04f22caae4ee31, State: Initialized, Role: FOLLOWER
I20260620 14:23:14.715600 21628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29d09b02e1449caa04f22caae4ee31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260620 14:23:14.715673 21628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:14.715700 21628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:14.715744 21628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:14.716337 21628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29d09b02e1449caa04f22caae4ee31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260620 14:23:14.716425 21628 leader_election.cc:302] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f29d09b02e1449caa04f22caae4ee31; no voters:
I20260620 14:23:14.716629 21628 leader_election.cc:288] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:14.716679 21633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:14.716863 21633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [term 1 LEADER]: Becoming Leader. State: Replica: 5f29d09b02e1449caa04f22caae4ee31, State: Running, Role: LEADER
I20260620 14:23:14.716933 21628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:14.717058 21633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29d09b02e1449caa04f22caae4ee31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260620 14:23:14.717480 21634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f29d09b02e1449caa04f22caae4ee31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29d09b02e1449caa04f22caae4ee31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } } }
I20260620 14:23:14.717561 21634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:14.718041 21641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:14.718622 21641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:23:14 dist-test-slave-h6n5 krb5kdc[21547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:23:14.718624 21635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f29d09b02e1449caa04f22caae4ee31. Latest consensus state: current_term: 1 leader_uuid: "5f29d09b02e1449caa04f22caae4ee31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29d09b02e1449caa04f22caae4ee31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } } }
I20260620 14:23:14.719933 21635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:14.720458 21641 catalog_manager.cc:1347] Generated new cluster ID: db8c0bf2ee604d7ca9e9dacc0e5114e1
I20260620 14:23:14.720511 21641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:14.723306 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:14.712357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:56639 (local address 127.0.0.1:41069)
0620 14:23:14.712531 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:14.712540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:14.712581 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:23:14.713035 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:14.713042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:14.713091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:14.713203 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:14.713990 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:14.714859 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:14.715969 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:14.716382 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:14.716536 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:14.719875 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:14.719879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:14.719879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:14.719892 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:14.721723 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:14.722135 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:14.722144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:14.722144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:14.722227 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:14.722476 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:14.722477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:14.722478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:14.722726 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:14.722901 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:14.723037 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:23:14.723142 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260620 14:23:14.728521 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:14.732676 21641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:14.733450 21641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:14.740227 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:14.729961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:36085 (local address 127.0.0.1:41069)
0620 14:23:14.730002 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:14.730004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:14.732823 (+ 2819us) server_negotiation.cc:407] Connection header received
0620 14:23:14.732888 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:14.732889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:14.732901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:14.732975 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:14.733640 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:14.734426 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:14.735389 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:14.735744 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:14.735828 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:14.737240 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:14.737241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:14.737242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:14.737249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:14.738919 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:14.739334 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:14.739335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:14.739336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:14.739400 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:14.739656 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:14.739657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:14.739657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:14.739797 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:14.739915 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:14.740059 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:23:14.740128 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:23:14.743986 21641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f29d09b02e1449caa04f22caae4ee31: Generated new TSK 0
I20260620 14:23:14.744187 21641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:14.746431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21560
W20260620 14:23:14.751021 21545 connection.cc:535] client connection to 127.0.0.1:41069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41069): Connection reset by peer (error 104)
2026-06-20T14:23:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:14 dist-test-slave-h6n5 krb5kdc[21659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:14 dist-test-slave-h6n5 krb5kdc[21659](info): set up 2 sockets
Jun 20 14:23:14 dist-test-slave-h6n5 krb5kdc[21659](info): commencing operation
krb5kdc: starting...
W20260620 14:23:15.779510 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:15 dist-test-slave-h6n5 krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:15.895687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:15.979918 21672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:15.980132 21672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:15.980158 21672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:15.981277 21672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:15.981314 21672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:15.981328 21672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:15.981340 21672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:15.981352 21672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:15.982767 21672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38649
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:15.982949 21672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:15.983147 21672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:15.988034 21677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:15.988076 21680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:15.988052 21672 server_base.cc:1034] running on GCE node
W20260620 14:23:15.988054 21678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:15.988515 21672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:15.988842 21672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:15.990011 21672 hybrid_clock.cc:648] HybridClock initialized: now 1781965395989985 us; error 36 us; skew 500 ppm
Jun 20 14:23:15 dist-test-slave-h6n5 krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:15.992467 21672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:15.992893 21672 webserver.cc:460] Webserver started at http://127.0.0.1:40883/ using document root <none> and password file <none>
I20260620 14:23:15.993145 21672 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:15.993222 21672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:15.993368 21672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:15.994467 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "5b177a9491044d4f9164823c4264d643"
format_stamp: "Formatted at 2026-06-20 14:23:15 on dist-test-slave-h6n5"
server_key: "84398005ffb46aec421f478f328d3930"
server_key_iv: "be47da066acc24ddd90b5bbbebc0bbd5"
server_key_version: "encryptionkey@0"
I20260620 14:23:15.994848 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b177a9491044d4f9164823c4264d643"
format_stamp: "Formatted at 2026-06-20 14:23:15 on dist-test-slave-h6n5"
server_key: "84398005ffb46aec421f478f328d3930"
server_key_iv: "be47da066acc24ddd90b5bbbebc0bbd5"
server_key_version: "encryptionkey@0"
I20260620 14:23:15.996291 21672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:15.997047 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:15.997241 21672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:15.997346 21672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "5b177a9491044d4f9164823c4264d643"
format_stamp: "Formatted at 2026-06-20 14:23:15 on dist-test-slave-h6n5"
server_key: "84398005ffb46aec421f478f328d3930"
server_key_iv: "be47da066acc24ddd90b5bbbebc0bbd5"
server_key_version: "encryptionkey@0"
I20260620 14:23:15.997419 21672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:16.010224 21672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:16.012264 21672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:16.012426 21672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:16.016394 21672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38649
I20260620 14:23:16.016428 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38649 every 8 connection(s)
I20260620 14:23:16.016796 21672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:16.017805 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:16.020206 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643: Bootstrap starting.
I20260620 14:23:16.020354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21672
I20260620 14:23:16.020462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:16.020684 392 external_mini_cluster.cc:1428] Setting key ae13aa2fd59e40c668356da518a7131a
I20260620 14:23:16.020951 21740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:16.021530 21740 log.cc:824] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:16.022300 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643: No bootstrap required, opened a new log
I20260620 14:23:16.024493 21740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b177a9491044d4f9164823c4264d643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260620 14:23:16.024658 21740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:16.024683 21740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b177a9491044d4f9164823c4264d643, State: Initialized, Role: FOLLOWER
I20260620 14:23:16.024814 21740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b177a9491044d4f9164823c4264d643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260620 14:23:16.024883 21740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:16.024904 21740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:16.024969 21740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:16.025724 21740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b177a9491044d4f9164823c4264d643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260620 14:23:16.025835 21740 leader_election.cc:302] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b177a9491044d4f9164823c4264d643; no voters:
I20260620 14:23:16.026011 21740 leader_election.cc:288] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:16.026077 21745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:16.026258 21740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:16.026311 21745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [term 1 LEADER]: Becoming Leader. State: Replica: 5b177a9491044d4f9164823c4264d643, State: Running, Role: LEADER
I20260620 14:23:16.026525 21745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b177a9491044d4f9164823c4264d643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260620 14:23:16.026984 21746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b177a9491044d4f9164823c4264d643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b177a9491044d4f9164823c4264d643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } } }
I20260620 14:23:16.027006 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:16.021162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:53493 (local address 127.0.0.1:38649)
0620 14:23:16.021351 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:16.021360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:16.021387 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:23:16.021878 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:16.021885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:16.021939 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:16.022060 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:16.022335 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:16.023267 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:16.024503 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:16.025116 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:16.025200 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:16.025962 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:16.025966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:16.025966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:16.026240 (+ 274us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:16.026282 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:16.026298 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:16.026718 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:16.026764 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:23:16.026791 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260620 14:23:16.027103 21746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:16.027256 21747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b177a9491044d4f9164823c4264d643. Latest consensus state: current_term: 1 leader_uuid: "5b177a9491044d4f9164823c4264d643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b177a9491044d4f9164823c4264d643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } } }
I20260620 14:23:16.027323 21747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643 [sys.catalog]: This master's current role is: LEADER
W20260620 14:23:16.028232 21761 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 14:23:16.028232 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 14:23:16.028285 21761 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:16.028345 21754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:16.028517 21754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:16.029928 21754 catalog_manager.cc:1347] Generated new cluster ID: c76b18c413b24281b1142b7b60b5cb29
I20260620 14:23:16.030000 21754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:16.035089 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:16.029698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:41137 (local address 127.0.0.1:38649)
0620 14:23:16.029739 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:16.029741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:16.029865 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:23:16.031718 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:16.031719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:16.031729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:16.031789 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:16.031936 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:16.032641 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:16.033725 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:16.034040 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:16.034098 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:16.034817 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:16.034818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:16.034818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:16.034823 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:16.034837 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:16.034841 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:16.034898 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:16.034999 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:23:16.035009 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:23:16.052027 21754 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:16.052618 21754 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:16.055598 21754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b177a9491044d4f9164823c4264d643: Generated new TSK 0
I20260620 14:23:16.055785 21754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:16.062672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21672
W20260620 14:23:16.066879 21657 connection.cc:535] client connection to 127.0.0.1:38649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38649): Connection reset by peer (error 104)
2026-06-20T14:23:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:16 dist-test-slave-h6n5 krb5kdc[21771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:16 dist-test-slave-h6n5 krb5kdc[21771](info): set up 2 sockets
Jun 20 14:23:16 dist-test-slave-h6n5 krb5kdc[21771](info): commencing operation
krb5kdc: starting...
W20260620 14:23:17.099738 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:17 dist-test-slave-h6n5 krb5kdc[21771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:23:17.214329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:17.295701 21784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:17.295909 21784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:17.295933 21784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:17.297047 21784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:17.297087 21784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:17.297101 21784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:17.297113 21784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:17.297125 21784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:17.298659 21784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:17.298859 21784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:17.299041 21784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:17.303952 21792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:17.303944 21789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:17.304139 21784 server_base.cc:1034] running on GCE node
W20260620 14:23:17.303944 21790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:17.304443 21784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:17.304750 21784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:17.305965 21784 hybrid_clock.cc:648] HybridClock initialized: now 1781965397305948 us; error 26 us; skew 500 ppm
Jun 20 14:23:17 dist-test-slave-h6n5 krb5kdc[21771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:17.308727 21784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:23:17.309136 21784 webserver.cc:460] Webserver started at http://127.0.0.1:34141/ using document root <none> and password file <none>
I20260620 14:23:17.309398 21784 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:17.309444 21784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:17.309602 21784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:17.310686 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "41efdef160d445b6bf5d9efd56db33da"
format_stamp: "Formatted at 2026-06-20 14:23:17 on dist-test-slave-h6n5"
server_key: "d5d36111cb0dd17f64fb57590bb59b3e"
server_key_iv: "5ec54edb94e7ad008835b1fc4142a507"
server_key_version: "encryptionkey@0"
I20260620 14:23:17.311053 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "41efdef160d445b6bf5d9efd56db33da"
format_stamp: "Formatted at 2026-06-20 14:23:17 on dist-test-slave-h6n5"
server_key: "d5d36111cb0dd17f64fb57590bb59b3e"
server_key_iv: "5ec54edb94e7ad008835b1fc4142a507"
server_key_version: "encryptionkey@0"
I20260620 14:23:17.312497 21784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:23:17.313236 21799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:17.313409 21784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:17.313526 21784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "41efdef160d445b6bf5d9efd56db33da"
format_stamp: "Formatted at 2026-06-20 14:23:17 on dist-test-slave-h6n5"
server_key: "d5d36111cb0dd17f64fb57590bb59b3e"
server_key_iv: "5ec54edb94e7ad008835b1fc4142a507"
server_key_version: "encryptionkey@0"
I20260620 14:23:17.313602 21784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:17.317049 21784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:17.319226 21784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:17.319377 21784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:17.323093 21784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43463
I20260620 14:23:17.323141 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43463 every 8 connection(s)
I20260620 14:23:17.323447 21784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:17.324442 21852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:17.326766 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da: Bootstrap starting.
I20260620 14:23:17.327358 21852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:17.327704 21852 log.cc:824] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:17.328385 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da: No bootstrap required, opened a new log
I20260620 14:23:17.328615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21784
I20260620 14:23:17.328727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:17.328928 392 external_mini_cluster.cc:1428] Setting key fff94b3be127fb554ed17d73219fb114
I20260620 14:23:17.330482 21852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41efdef160d445b6bf5d9efd56db33da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260620 14:23:17.330651 21852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:17.330729 21852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41efdef160d445b6bf5d9efd56db33da, State: Initialized, Role: FOLLOWER
I20260620 14:23:17.331218 21852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41efdef160d445b6bf5d9efd56db33da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260620 14:23:17.331349 21852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:17.331386 21852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:17.331436 21852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:17.331939 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:17.329415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:42371 (local address 127.0.0.1:43463)
0620 14:23:17.329575 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:17.329583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:17.329606 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:23:17.330488 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:17.330496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:17.330528 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:17.330706 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:17.331366 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:17.331370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:17.331370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:17.331375 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:17.331423 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:17.331445 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:17.331544 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:17.331669 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:23:17.331730 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260620 14:23:17.332598 21852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41efdef160d445b6bf5d9efd56db33da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260620 14:23:17.332710 21852 leader_election.cc:302] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41efdef160d445b6bf5d9efd56db33da; no voters:
I20260620 14:23:17.332885 21852 leader_election.cc:288] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:17.332917 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:17.332969 21857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:17.333127 21852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:17.333138 21857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [term 1 LEADER]: Becoming Leader. State: Replica: 41efdef160d445b6bf5d9efd56db33da, State: Running, Role: LEADER
I20260620 14:23:17.333345 21857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41efdef160d445b6bf5d9efd56db33da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260620 14:23:17.333702 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41efdef160d445b6bf5d9efd56db33da. Latest consensus state: current_term: 1 leader_uuid: "41efdef160d445b6bf5d9efd56db33da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41efdef160d445b6bf5d9efd56db33da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } } }
I20260620 14:23:17.333911 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:17.333840 21859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41efdef160d445b6bf5d9efd56db33da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41efdef160d445b6bf5d9efd56db33da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } } }
I20260620 14:23:17.334352 21859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:17.334565 21868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:17.335476 21868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:17.335785 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:17.334468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.12:42293 (local address 127.0.0.1:43463)
0620 14:23:17.334506 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:17.334509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:17.334798 (+ 289us) server_negotiation.cc:407] Connection header received
0620 14:23:17.334864 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:17.334865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:17.334871 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:17.334925 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:17.335468 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:17.335470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:17.335470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:17.335475 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:17.335490 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:17.335495 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:17.335608 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:17.335710 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:23:17.335724 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:23:17.337137 21868 catalog_manager.cc:1347] Generated new cluster ID: a1338cd0dbe74e779683cd20d0ca5cd8
I20260620 14:23:17.337189 21868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:17.346688 21868 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:17.346863 21868 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:17.354188 21868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41efdef160d445b6bf5d9efd56db33da: Generated new TSK 0
I20260620 14:23:17.354370 21868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:17.358410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21784
2026-06-20T14:23:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:17 dist-test-slave-h6n5 krb5kdc[21883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:17 dist-test-slave-h6n5 krb5kdc[21883](info): set up 2 sockets
Jun 20 14:23:17 dist-test-slave-h6n5 krb5kdc[21883](info): commencing operation
krb5kdc: starting...
W20260620 14:23:18.403642 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:18 dist-test-slave-h6n5 krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:18.517680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36261
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:18.597522 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:18.597725 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:18.597748 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:18.598940 21896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:18.598986 21896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:18.599000 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:18.599045 21896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:18.599073 21896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:18.600592 21896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:18.600847 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:18.601074 21896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:18.605866 21901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:18.605906 21902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:18.605866 21904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:18.606166 21896 server_base.cc:1034] running on GCE node
I20260620 14:23:18.606407 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:18.606654 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:18.607824 21896 hybrid_clock.cc:648] HybridClock initialized: now 1781965398607796 us; error 44 us; skew 500 ppm
Jun 20 14:23:18 dist-test-slave-h6n5 krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:18.610448 21896 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:18.610947 21896 webserver.cc:460] Webserver started at http://127.0.98.62:42061/ using document root <none> and password file <none>
I20260620 14:23:18.611182 21896 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:18.611248 21896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:18.611402 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:18.612553 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "a21959541abe4c468617ac0ba8a20e1f"
format_stamp: "Formatted at 2026-06-20 14:23:18 on dist-test-slave-h6n5"
server_key: "0113888f08917d4f574b2390e843582c"
server_key_iv: "185354e04a3af5d2269cedf5c1a85e90"
server_key_version: "encryptionkey@0"
I20260620 14:23:18.612915 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "a21959541abe4c468617ac0ba8a20e1f"
format_stamp: "Formatted at 2026-06-20 14:23:18 on dist-test-slave-h6n5"
server_key: "0113888f08917d4f574b2390e843582c"
server_key_iv: "185354e04a3af5d2269cedf5c1a85e90"
server_key_version: "encryptionkey@0"
I20260620 14:23:18.614246 21896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:23:18.614989 21911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:18.615170 21896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:18.615265 21896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "a21959541abe4c468617ac0ba8a20e1f"
format_stamp: "Formatted at 2026-06-20 14:23:18 on dist-test-slave-h6n5"
server_key: "0113888f08917d4f574b2390e843582c"
server_key_iv: "185354e04a3af5d2269cedf5c1a85e90"
server_key_version: "encryptionkey@0"
I20260620 14:23:18.615355 21896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:18.626398 21896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:18.628696 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:18.628858 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:18.632896 21896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36261
I20260620 14:23:18.632930 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36261 every 8 connection(s)
I20260620 14:23:18.633293 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:18.634121 21964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:18.636436 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f: Bootstrap starting.
I20260620 14:23:18.636978 21964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:18.637208 21964 log.cc:824] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:18.637895 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f: No bootstrap required, opened a new log
I20260620 14:23:18.639458 21964 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21959541abe4c468617ac0ba8a20e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36261 } }
I20260620 14:23:18.639766 21964 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:18.639798 21964 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21959541abe4c468617ac0ba8a20e1f, State: Initialized, Role: FOLLOWER
I20260620 14:23:18.639936 21964 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21959541abe4c468617ac0ba8a20e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36261 } }
I20260620 14:23:18.640017 21964 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:18.640039 21964 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:18.640067 21964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:18.640563 21964 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21959541abe4c468617ac0ba8a20e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36261 } }
I20260620 14:23:18.640646 21964 leader_election.cc:302] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21959541abe4c468617ac0ba8a20e1f; no voters:
I20260620 14:23:18.640774 21964 leader_election.cc:288] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:18.640827 21967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:18.640990 21967 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [term 1 LEADER]: Becoming Leader. State: Replica: a21959541abe4c468617ac0ba8a20e1f, State: Running, Role: LEADER
I20260620 14:23:18.641027 21964 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:18.641099 21967 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21959541abe4c468617ac0ba8a20e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36261 } }
I20260620 14:23:18.641515 21969 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21959541abe4c468617ac0ba8a20e1f. Latest consensus state: current_term: 1 leader_uuid: "a21959541abe4c468617ac0ba8a20e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21959541abe4c468617ac0ba8a20e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36261 } } }
I20260620 14:23:18.641640 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a21959541abe4c468617ac0ba8a20e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21959541abe4c468617ac0ba8a20e1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36261 } } }
I20260620 14:23:18.641657 21969 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:18.641778 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:18.642108 21976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:18.642395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 21896
I20260620 14:23:18.642530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:18.642737 392 external_mini_cluster.cc:1428] Setting key 2b39a2a522bb57657d6109bac2697206
I20260620 14:23:18.643406 21976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:18.644830 21976 catalog_manager.cc:1347] Generated new cluster ID: c484627439e545ca8512759141776153
I20260620 14:23:18.644886 21976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:23:18 dist-test-slave-h6n5 krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:18.656765 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:18.643253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33125 (local address 127.0.98.62:36261)
0620 14:23:18.643391 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:18.643402 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:18.647831 (+ 4429us) server_negotiation.cc:407] Connection header received
0620 14:23:18.647981 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:18.647986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:18.648029 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:18.648157 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:18.648796 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:18.649696 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:18.650465 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:18.650860 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:18.650975 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:18.653331 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:18.653335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:18.653335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:18.653353 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:18.655016 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:18.655502 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:18.655505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:18.655505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:18.655660 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:18.655893 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:18.655894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:18.655895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:18.656136 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:18.656311 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:18.656394 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:23:18.656500 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260620 14:23:18.657171 21976 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:18.657744 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:18.657799 21976 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:18.666832 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:18.659184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45611 (local address 127.0.98.62:36261)
0620 14:23:18.659218 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:18.659219 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:18.659698 (+ 479us) server_negotiation.cc:407] Connection header received
0620 14:23:18.659727 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:18.659728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:18.659739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:18.659810 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:18.660382 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:18.661160 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:18.662157 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:18.662506 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:18.662578 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:18.663906 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:18.663908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:18.663908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:18.663915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:18.665604 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:18.666019 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:18.666021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:18.666021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:18.666090 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:18.666333 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:18.666334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:18.666334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:18.666460 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:18.666574 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:18.666675 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:23:18.666740 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:23:18.668793 21976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21959541abe4c468617ac0ba8a20e1f: Generated new TSK 0
I20260620 14:23:18.668991 21976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:18.674263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 21896
W20260620 14:23:18.678515 21881 connection.cc:535] client connection to 127.0.98.62:36261 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36261): Connection reset by peer (error 104)
2026-06-20T14:23:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:18 dist-test-slave-h6n5 krb5kdc[21995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:18 dist-test-slave-h6n5 krb5kdc[21995](info): set up 2 sockets
Jun 20 14:23:18 dist-test-slave-h6n5 krb5kdc[21995](info): commencing operation
krb5kdc: starting...
W20260620 14:23:19.703832 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:19 dist-test-slave-h6n5 krb5kdc[21995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:19.820890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:19.901047 22008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:19.901296 22008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:19.901337 22008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:19.902494 22008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:19.902566 22008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:19.902590 22008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:19.902617 22008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:19.902649 22008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:19.904104 22008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:19.904395 22008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:19.904631 22008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:19.909857 22014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:19.909865 22016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:19.910033 22008 server_base.cc:1034] running on GCE node
W20260620 14:23:19.909865 22013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:19.910382 22008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:19.910650 22008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:19.911815 22008 hybrid_clock.cc:648] HybridClock initialized: now 1781965399911794 us; error 34 us; skew 500 ppm
Jun 20 14:23:19 dist-test-slave-h6n5 krb5kdc[21995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:19.918828 22008 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:19.919256 22008 webserver.cc:460] Webserver started at http://127.0.98.62:34721/ using document root <none> and password file <none>
I20260620 14:23:19.919445 22008 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:19.919486 22008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:19.919658 22008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:19.920724 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "8ca23b69b6fe463695e6ffea539e425b"
format_stamp: "Formatted at 2026-06-20 14:23:19 on dist-test-slave-h6n5"
server_key: "e830a1fac576f6c266a2c27952ac42fc"
server_key_iv: "617783ef7383020bf882c13f3c643a17"
server_key_version: "encryptionkey@0"
I20260620 14:23:19.921042 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ca23b69b6fe463695e6ffea539e425b"
format_stamp: "Formatted at 2026-06-20 14:23:19 on dist-test-slave-h6n5"
server_key: "e830a1fac576f6c266a2c27952ac42fc"
server_key_iv: "617783ef7383020bf882c13f3c643a17"
server_key_version: "encryptionkey@0"
I20260620 14:23:19.922360 22008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:23:19.923071 22023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:19.923260 22008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:19.923362 22008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "8ca23b69b6fe463695e6ffea539e425b"
format_stamp: "Formatted at 2026-06-20 14:23:19 on dist-test-slave-h6n5"
server_key: "e830a1fac576f6c266a2c27952ac42fc"
server_key_iv: "617783ef7383020bf882c13f3c643a17"
server_key_version: "encryptionkey@0"
I20260620 14:23:19.923445 22008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:19.932888 22008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:19.934969 22008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:19.935139 22008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:19.939265 22008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44881
I20260620 14:23:19.939303 22075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44881 every 8 connection(s)
I20260620 14:23:19.939746 22008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:19.940662 22076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:19.943075 22076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b: Bootstrap starting.
I20260620 14:23:19.943755 22076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:19.944046 22076 log.cc:824] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:19.944757 22076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b: No bootstrap required, opened a new log
I20260620 14:23:19.945696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22008
I20260620 14:23:19.945823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:19.946033 392 external_mini_cluster.cc:1428] Setting key c21a8bd0ef5cdce84c88e853788668d6
I20260620 14:23:19.947018 22076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca23b69b6fe463695e6ffea539e425b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44881 } }
I20260620 14:23:19.947196 22076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:19.947237 22076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca23b69b6fe463695e6ffea539e425b, State: Initialized, Role: FOLLOWER
I20260620 14:23:19.947378 22076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca23b69b6fe463695e6ffea539e425b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44881 } }
I20260620 14:23:19.947470 22076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:19.947503 22076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:19.947585 22076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:19.948359 22076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca23b69b6fe463695e6ffea539e425b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44881 } }
I20260620 14:23:19.948467 22076 leader_election.cc:302] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ca23b69b6fe463695e6ffea539e425b; no voters:
I20260620 14:23:19.948618 22076 leader_election.cc:288] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:19.948680 22081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:19.948848 22081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [term 1 LEADER]: Becoming Leader. State: Replica: 8ca23b69b6fe463695e6ffea539e425b, State: Running, Role: LEADER
I20260620 14:23:19.948868 22076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:19.948977 22081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca23b69b6fe463695e6ffea539e425b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44881 } }
I20260620 14:23:19.949327 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ca23b69b6fe463695e6ffea539e425b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca23b69b6fe463695e6ffea539e425b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44881 } } }
I20260620 14:23:19.949319 22081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca23b69b6fe463695e6ffea539e425b. Latest consensus state: current_term: 1 leader_uuid: "8ca23b69b6fe463695e6ffea539e425b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca23b69b6fe463695e6ffea539e425b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44881 } } }
I20260620 14:23:19.949467 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:19.949476 22081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:19.949802 22088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:19.950577 22088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:19.952234 22088 catalog_manager.cc:1347] Generated new cluster ID: 6241c85d231948788413e0e0929abcc1
I20260620 14:23:19.952292 22088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:19.952785 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:19.946550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60875 (local address 127.0.98.62:44881)
0620 14:23:19.946730 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:19.946739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:19.946781 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:23:19.946979 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:19.946984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:19.947035 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:19.947145 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:19.947436 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:19.949198 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:19.950457 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:19.951077 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:19.951202 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:19.952053 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:19.952058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:19.952058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:19.952295 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:19.952335 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:19.952354 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:19.952456 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:19.952537 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:23:19.952568 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260620 14:23:19.953676 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:19.958829 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:19.955128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60319 (local address 127.0.98.62:44881)
0620 14:23:19.955167 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:19.955168 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:19.955941 (+ 773us) server_negotiation.cc:407] Connection header received
0620 14:23:19.956017 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:19.956017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:19.956030 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:19.956088 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:19.956244 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:19.956741 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:19.957452 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:19.957680 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:19.957769 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:19.958504 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:19.958505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:19.958506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:19.958510 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:19.958525 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:19.958529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:19.958561 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:19.958705 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:23:19.958720 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:23:19.977950 22088 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:19.978669 22088 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:19.990890 22088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ca23b69b6fe463695e6ffea539e425b: Generated new TSK 0
I20260620 14:23:19.991106 22088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:19.995525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22008
W20260620 14:23:19.999607 21993 connection.cc:535] client connection to 127.0.98.62:44881 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44881): Connection reset by peer (error 104)
2026-06-20T14:23:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:20 dist-test-slave-h6n5 krb5kdc[22107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:20 dist-test-slave-h6n5 krb5kdc[22107](info): set up 2 sockets
Jun 20 14:23:20 dist-test-slave-h6n5 krb5kdc[22107](info): commencing operation
krb5kdc: starting...
W20260620 14:23:21.039784 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:21 dist-test-slave-h6n5 krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:21.154903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:21.236094 22120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:21.236326 22120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:21.236364 22120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:21.237500 22120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:21.237566 22120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:21.237593 22120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:21.237620 22120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:21.237648 22120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:21.239102 22120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:21.239372 22120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:21.239651 22120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:21.244359 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:21.244545 22120 server_base.cc:1034] running on GCE node
W20260620 14:23:21.244372 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:21.244352 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:21.244839 22120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:21.245129 22120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:21.246356 22120 hybrid_clock.cc:648] HybridClock initialized: now 1781965401246332 us; error 34 us; skew 500 ppm
Jun 20 14:23:21 dist-test-slave-h6n5 krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:21.248950 22120 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:21.249387 22120 webserver.cc:460] Webserver started at http://127.0.98.62:35213/ using document root <none> and password file <none>
I20260620 14:23:21.249635 22120 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:21.249706 22120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:21.249850 22120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:21.250901 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "e9f12b46c7094e87a073d9b81363dd3e"
format_stamp: "Formatted at 2026-06-20 14:23:21 on dist-test-slave-h6n5"
server_key: "f036c1ff485d776299a6848756ec78f3"
server_key_iv: "cc05eb0514176692f6d81731db498a00"
server_key_version: "encryptionkey@0"
I20260620 14:23:21.251264 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9f12b46c7094e87a073d9b81363dd3e"
format_stamp: "Formatted at 2026-06-20 14:23:21 on dist-test-slave-h6n5"
server_key: "f036c1ff485d776299a6848756ec78f3"
server_key_iv: "cc05eb0514176692f6d81731db498a00"
server_key_version: "encryptionkey@0"
I20260620 14:23:21.252688 22120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:23:21.253407 22135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:21.253597 22120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:21.253700 22120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "e9f12b46c7094e87a073d9b81363dd3e"
format_stamp: "Formatted at 2026-06-20 14:23:21 on dist-test-slave-h6n5"
server_key: "f036c1ff485d776299a6848756ec78f3"
server_key_iv: "cc05eb0514176692f6d81731db498a00"
server_key_version: "encryptionkey@0"
I20260620 14:23:21.253783 22120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:21.257109 22120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:21.259241 22120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:21.259413 22120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:21.263319 22120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43613
I20260620 14:23:21.263387 22187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43613 every 8 connection(s)
I20260620 14:23:21.263775 22120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:21.264750 22188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:21.266917 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e: Bootstrap starting.
I20260620 14:23:21.267426 22188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:21.267735 22188 log.cc:824] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:21.268365 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e: No bootstrap required, opened a new log
I20260620 14:23:21.269170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22120
I20260620 14:23:21.269294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:21.269469 392 external_mini_cluster.cc:1428] Setting key da1cebd562775d48b38caead7cc652d9
I20260620 14:23:21.270522 22188 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f12b46c7094e87a073d9b81363dd3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43613 } }
I20260620 14:23:21.270692 22188 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:21.270735 22188 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f12b46c7094e87a073d9b81363dd3e, State: Initialized, Role: FOLLOWER
I20260620 14:23:21.270952 22188 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f12b46c7094e87a073d9b81363dd3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43613 } }
I20260620 14:23:21.271044 22188 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:21.271068 22188 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:21.271095 22188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:21.271926 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:21.269982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54189 (local address 127.0.98.62:43613)
0620 14:23:21.270189 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:21.270196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:21.270220 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:23:21.270503 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:21.270509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:21.270537 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:21.270651 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:21.271399 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:21.271402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:21.271402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:21.271406 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:21.271460 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:21.271471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:21.271526 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:21.271674 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:23:21.271735 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
I20260620 14:23:21.272234 22188 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f12b46c7094e87a073d9b81363dd3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43613 } }
I20260620 14:23:21.272352 22188 leader_election.cc:302] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f12b46c7094e87a073d9b81363dd3e; no voters:
I20260620 14:23:21.272557 22188 leader_election.cc:288] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:21.272567 22193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:21.272759 22193 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [term 1 LEADER]: Becoming Leader. State: Replica: e9f12b46c7094e87a073d9b81363dd3e, State: Running, Role: LEADER
I20260620 14:23:21.272840 22188 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:21.272856 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:21.272892 22193 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f12b46c7094e87a073d9b81363dd3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43613 } }
I20260620 14:23:21.273248 22194 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f12b46c7094e87a073d9b81363dd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f12b46c7094e87a073d9b81363dd3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43613 } } }
I20260620 14:23:21.273265 22195 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f12b46c7094e87a073d9b81363dd3e. Latest consensus state: current_term: 1 leader_uuid: "e9f12b46c7094e87a073d9b81363dd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f12b46c7094e87a073d9b81363dd3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43613 } } }
I20260620 14:23:21.273355 22194 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:21.273367 22195 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:21.273553 22201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:21.274127 22201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:21.275776 22201 catalog_manager.cc:1347] Generated new cluster ID: 695302b4613a409f872e14a855319bfa
I20260620 14:23:21.275847 22201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:21.276904 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:21.274363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45537 (local address 127.0.98.62:43613)
0620 14:23:21.274391 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:21.274394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:21.275899 (+ 1505us) server_negotiation.cc:407] Connection header received
0620 14:23:21.276003 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:21.276004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:21.276015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:21.276081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:21.276688 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:21.276690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:21.276690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:21.276693 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:21.276707 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:21.276711 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:21.276763 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:21.276837 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:23:21.276849 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11}
I20260620 14:23:21.298976 22201 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:21.299186 22201 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:21.302682 22201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f12b46c7094e87a073d9b81363dd3e: Generated new TSK 0
I20260620 14:23:21.302876 22201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:21.305114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22120
2026-06-20T14:23:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:21 dist-test-slave-h6n5 krb5kdc[22219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:21 dist-test-slave-h6n5 krb5kdc[22219](info): set up 2 sockets
Jun 20 14:23:21 dist-test-slave-h6n5 krb5kdc[22219](info): commencing operation
krb5kdc: starting...
W20260620 14:23:22.339600 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:22 dist-test-slave-h6n5 krb5kdc[22219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:22.453909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34919
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:22.533663 22232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:22.533857 22232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:22.533879 22232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:22.535019 22232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:22.535059 22232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:22.535074 22232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:22.535116 22232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:22.535146 22232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:22.536581 22232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:22.536824 22232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:22.537043 22232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:22.541749 22237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:22.541746 22238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:22.541903 22232 server_base.cc:1034] running on GCE node
W20260620 14:23:22.541734 22240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:22.542243 22232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:22.542523 22232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:22.543694 22232 hybrid_clock.cc:648] HybridClock initialized: now 1781965402543682 us; error 46 us; skew 500 ppm
Jun 20 14:23:22 dist-test-slave-h6n5 krb5kdc[22219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:22.546227 22232 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:22.546675 22232 webserver.cc:460] Webserver started at http://127.0.98.62:43589/ using document root <none> and password file <none>
I20260620 14:23:22.546896 22232 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:22.546941 22232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:22.547088 22232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:22.548228 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "58a105723e9b4eed9e2ef874a281dc4d"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-h6n5"
server_key: "5fafbf747b5d2f379a4637d705d21d31"
server_key_iv: "984323be02bf50945aba084a4fc9c8c4"
server_key_version: "encryptionkey@0"
I20260620 14:23:22.548592 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "58a105723e9b4eed9e2ef874a281dc4d"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-h6n5"
server_key: "5fafbf747b5d2f379a4637d705d21d31"
server_key_iv: "984323be02bf50945aba084a4fc9c8c4"
server_key_version: "encryptionkey@0"
I20260620 14:23:22.549906 22232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:23:22.550704 22247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:22.550904 22232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:22.550987 22232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "58a105723e9b4eed9e2ef874a281dc4d"
format_stamp: "Formatted at 2026-06-20 14:23:22 on dist-test-slave-h6n5"
server_key: "5fafbf747b5d2f379a4637d705d21d31"
server_key_iv: "984323be02bf50945aba084a4fc9c8c4"
server_key_version: "encryptionkey@0"
I20260620 14:23:22.551043 22232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:22.571143 22232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:22.573526 22232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:22.573652 22232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:22.577405 22232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34919
I20260620 14:23:22.577489 22299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34919 every 8 connection(s)
I20260620 14:23:22.577836 22232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:22.578707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22232
I20260620 14:23:22.578804 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:22.578838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:22.579048 392 external_mini_cluster.cc:1428] Setting key 7585955e5177051db06c1dfd2ff8371b
I20260620 14:23:22.581974 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d: Bootstrap starting.
I20260620 14:23:22.582695 22300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:22.582995 22300 log.cc:824] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:22.583778 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d: No bootstrap required, opened a new log
Jun 20 14:23:22 dist-test-slave-h6n5 krb5kdc[22219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:22.585984 22300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a105723e9b4eed9e2ef874a281dc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34919 } }
I20260620 14:23:22.586180 22300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:22.586226 22300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a105723e9b4eed9e2ef874a281dc4d, State: Initialized, Role: FOLLOWER
I20260620 14:23:22.586355 22300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a105723e9b4eed9e2ef874a281dc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34919 } }
I20260620 14:23:22.586444 22300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:22.586478 22300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:22.586532 22300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:22.587188 22300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a105723e9b4eed9e2ef874a281dc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34919 } }
I20260620 14:23:22.587288 22300 leader_election.cc:302] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a105723e9b4eed9e2ef874a281dc4d; no voters:
I20260620 14:23:22.587492 22300 leader_election.cc:288] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:22.587520 22305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:22.587790 22300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:22.587913 22305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [term 1 LEADER]: Becoming Leader. State: Replica: 58a105723e9b4eed9e2ef874a281dc4d, State: Running, Role: LEADER
I20260620 14:23:22.588070 22305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a105723e9b4eed9e2ef874a281dc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34919 } }
I20260620 14:23:22.588485 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a105723e9b4eed9e2ef874a281dc4d. Latest consensus state: current_term: 1 leader_uuid: "58a105723e9b4eed9e2ef874a281dc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a105723e9b4eed9e2ef874a281dc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34919 } } }
I20260620 14:23:22.588495 22306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a105723e9b4eed9e2ef874a281dc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a105723e9b4eed9e2ef874a281dc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34919 } } }
I20260620 14:23:22.588658 22306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:22.588855 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d [sys.catalog]: This master's current role is: LEADER
W20260620 14:23:22.589602 22320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:22.589660 22320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:22.589709 22314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:22.589852 22314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:22.590689 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:22.579634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35321 (local address 127.0.98.62:34919)
0620 14:23:22.579807 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:22.579814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:22.579839 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:23:22.580295 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:22.580302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:22.580349 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:22.580461 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:22.581254 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:22.582205 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:22.583312 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:22.583861 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:22.583999 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:22.586652 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:22.586657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:22.586657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:22.586674 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:22.588785 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:22.589219 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:22.589223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:22.589223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:22.589327 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:22.589562 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:22.589563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:22.589563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:22.589909 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:22.590049 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:22.590306 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:23:22.590417 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260620 14:23:22.591521 22314 catalog_manager.cc:1347] Generated new cluster ID: 67f51397950e411c9ab0fc03282672d3
I20260620 14:23:22.591596 22314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:22.591830 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:22.599174 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:22.593155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37355 (local address 127.0.98.62:34919)
0620 14:23:22.593205 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:22.593207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:22.593322 (+ 115us) server_negotiation.cc:407] Connection header received
0620 14:23:22.593394 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:22.593395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:22.593405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:22.593467 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:22.593963 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:22.594440 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:22.595133 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:22.595375 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:22.595468 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:22.596651 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:22.596652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:22.596652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:22.596659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:22.598060 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:22.598426 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:22.598427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:22.598427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:22.598480 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:22.598682 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:22.598683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:22.598683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:22.598796 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:22.598898 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:22.599004 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:23:22.599074 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:23:22.602126 22314 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:22.602643 22314 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:22.613067 22314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a105723e9b4eed9e2ef874a281dc4d: Generated new TSK 0
I20260620 14:23:22.613255 22314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:22.618680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22232
2026-06-20T14:23:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:22 dist-test-slave-h6n5 krb5kdc[22331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:22 dist-test-slave-h6n5 krb5kdc[22331](info): set up 2 sockets
Jun 20 14:23:22 dist-test-slave-h6n5 krb5kdc[22331](info): commencing operation
krb5kdc: starting...
W20260620 14:23:23.647600 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:23 dist-test-slave-h6n5 krb5kdc[22331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:23.762271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:23.842905 22344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:23.843145 22344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:23.843181 22344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:23.844352 22344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:23.844429 22344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:23.844456 22344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:23.844487 22344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:23.844521 22344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:23.845971 22344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:23.846239 22344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:23.846463 22344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:23.851051 22352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:23.851132 22349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:23.851061 22350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:23.851536 22344 server_base.cc:1034] running on GCE node
I20260620 14:23:23.851821 22344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:23.852092 22344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:23.853255 22344 hybrid_clock.cc:648] HybridClock initialized: now 1781965403853239 us; error 46 us; skew 500 ppm
Jun 20 14:23:23 dist-test-slave-h6n5 krb5kdc[22331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:23.855875 22344 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:23.856344 22344 webserver.cc:460] Webserver started at http://127.0.98.62:41429/ using document root <none> and password file <none>
I20260620 14:23:23.856573 22344 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:23.856642 22344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:23.856797 22344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:23.857810 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "e9d6a932a13345c4a439c119ba546f9d"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-h6n5"
server_key: "d4bd850ab350649d4dac6f950e8c3722"
server_key_iv: "579cdd887fe16f8f748e1d01c72b26bf"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.858170 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9d6a932a13345c4a439c119ba546f9d"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-h6n5"
server_key: "d4bd850ab350649d4dac6f950e8c3722"
server_key_iv: "579cdd887fe16f8f748e1d01c72b26bf"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.859457 22344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:23.860199 22359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.860378 22344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:23.860504 22344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "e9d6a932a13345c4a439c119ba546f9d"
format_stamp: "Formatted at 2026-06-20 14:23:23 on dist-test-slave-h6n5"
server_key: "d4bd850ab350649d4dac6f950e8c3722"
server_key_iv: "579cdd887fe16f8f748e1d01c72b26bf"
server_key_version: "encryptionkey@0"
I20260620 14:23:23.860589 22344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:23.889760 22344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:23.891871 22344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:23.892045 22344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:23.896153 22344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43601
I20260620 14:23:23.896235 22411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43601 every 8 connection(s)
I20260620 14:23:23.896588 22344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:23.897433 22412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:23.899721 22412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d: Bootstrap starting.
I20260620 14:23:23.900245 22412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:23.900475 22412 log.cc:824] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:23.901098 22412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d: No bootstrap required, opened a new log
I20260620 14:23:23.902658 22412 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a932a13345c4a439c119ba546f9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260620 14:23:23.902799 22412 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:23.902822 22412 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d6a932a13345c4a439c119ba546f9d, State: Initialized, Role: FOLLOWER
I20260620 14:23:23.902918 22412 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a932a13345c4a439c119ba546f9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260620 14:23:23.902968 22412 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:23.902987 22412 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:23.903059 22412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:23.903606 22412 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a932a13345c4a439c119ba546f9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260620 14:23:23.903723 22412 leader_election.cc:302] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d6a932a13345c4a439c119ba546f9d; no voters:
I20260620 14:23:23.903903 22412 leader_election.cc:288] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:23.903923 22415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:23.904115 22415 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [term 1 LEADER]: Becoming Leader. State: Replica: e9d6a932a13345c4a439c119ba546f9d, State: Running, Role: LEADER
I20260620 14:23:23.904177 22412 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:23.904229 22415 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a932a13345c4a439c119ba546f9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } }
I20260620 14:23:23.904619 22417 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d6a932a13345c4a439c119ba546f9d. Latest consensus state: current_term: 1 leader_uuid: "e9d6a932a13345c4a439c119ba546f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a932a13345c4a439c119ba546f9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } } }
I20260620 14:23:23.904749 22416 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d6a932a13345c4a439c119ba546f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a932a13345c4a439c119ba546f9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43601 } } }
I20260620 14:23:23.904759 22417 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:23.904855 22416 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:23.905221 22424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:23.905748 22424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:23.906798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22344
I20260620 14:23:23.906927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:23.907127 392 external_mini_cluster.cc:1428] Setting key fe97af20997a4eb7678645bf24a61d08
I20260620 14:23:23.907464 22424 catalog_manager.cc:1347] Generated new cluster ID: 7c5409c2037e4f11b260f0b4940fd7ef
I20260620 14:23:23.907514 22424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:23.913734 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.907771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35089 (local address 127.0.98.62:43601)
0620 14:23:23.907962 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.907972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.908014 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:23:23.908132 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.908139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.908192 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.908336 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:23.908556 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.910024 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.911115 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.911789 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.911897 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.912752 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:23.912756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:23.912756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:23.913122 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:23.913165 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:23.913207 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:23.913267 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.913394 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.913495 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260620 14:23:23.914589 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:23.917323 22424 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:23.917974 22424 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:23.920795 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:23.916468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40397 (local address 127.0.98.62:43601)
0620 14:23:23.916502 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:23.916504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:23.916512 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:23:23.916569 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:23.916570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:23.916580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:23.917203 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:23.917586 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.918270 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.919340 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:23.919657 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:23.919702 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:23.920524 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:23.920525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:23.920525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:23.920531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:23.920544 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:23.920548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:23.920604 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:23.920661 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:23:23.920699 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:23:23.921517 22424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d6a932a13345c4a439c119ba546f9d: Generated new TSK 0
I20260620 14:23:23.921676 22424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:23.925302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22344
2026-06-20T14:23:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:23 dist-test-slave-h6n5 krb5kdc[22443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:23 dist-test-slave-h6n5 krb5kdc[22443](info): set up 2 sockets
Jun 20 14:23:23 dist-test-slave-h6n5 krb5kdc[22443](info): commencing operation
krb5kdc: starting...
W20260620 14:23:24.955720 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:25 dist-test-slave-h6n5 krb5kdc[22443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:25.072196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:25.152508 22456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:25.152738 22456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:25.152765 22456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:25.153941 22456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:25.153982 22456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:25.153996 22456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:25.154009 22456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:25.154021 22456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:25.155514 22456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:25.155778 22456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:25.156006 22456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:25.160796 22461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:25.160743 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:25.160738 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:25.161031 22456 server_base.cc:1034] running on GCE node
I20260620 14:23:25.161259 22456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:25.161499 22456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:25.162647 22456 hybrid_clock.cc:648] HybridClock initialized: now 1781965405162632 us; error 27 us; skew 500 ppm
Jun 20 14:23:25 dist-test-slave-h6n5 krb5kdc[22443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:25.171430 22456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:25.172008 22456 webserver.cc:460] Webserver started at http://127.0.98.62:40659/ using document root <none> and password file <none>
I20260620 14:23:25.172245 22456 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:25.172314 22456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:25.172487 22456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:25.173616 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "21fae5ef37b143ed9a1828e9f1eb4c30"
format_stamp: "Formatted at 2026-06-20 14:23:25 on dist-test-slave-h6n5"
server_key: "d0b1675f76b84c80bac5c556914635cc"
server_key_iv: "26561aab7fddd6b53ef34553390dbd4b"
server_key_version: "encryptionkey@0"
I20260620 14:23:25.173998 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "21fae5ef37b143ed9a1828e9f1eb4c30"
format_stamp: "Formatted at 2026-06-20 14:23:25 on dist-test-slave-h6n5"
server_key: "d0b1675f76b84c80bac5c556914635cc"
server_key_iv: "26561aab7fddd6b53ef34553390dbd4b"
server_key_version: "encryptionkey@0"
I20260620 14:23:25.175362 22456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:23:25.176265 22471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:25.176469 22456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:25.176574 22456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "21fae5ef37b143ed9a1828e9f1eb4c30"
format_stamp: "Formatted at 2026-06-20 14:23:25 on dist-test-slave-h6n5"
server_key: "d0b1675f76b84c80bac5c556914635cc"
server_key_iv: "26561aab7fddd6b53ef34553390dbd4b"
server_key_version: "encryptionkey@0"
I20260620 14:23:25.176666 22456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:25.180131 22456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:25.182746 22456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:25.182921 22456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:25.186969 22456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39861
I20260620 14:23:25.187000 22523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39861 every 8 connection(s)
I20260620 14:23:25.187336 22456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:25.188323 22524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:25.190567 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30: Bootstrap starting.
I20260620 14:23:25.191123 22524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:25.191479 22524 log.cc:824] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:25.192382 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30: No bootstrap required, opened a new log
I20260620 14:23:25.193993 22524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260620 14:23:25.194130 22524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:23:25.194154 22524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fae5ef37b143ed9a1828e9f1eb4c30, State: Initialized, Role: FOLLOWER
I20260620 14:23:25.194310 22524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260620 14:23:25.194393 22524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:23:25.194419 22524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:23:25.194470 22524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:25.195006 22524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260620 14:23:25.195092 22524 leader_election.cc:302] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21fae5ef37b143ed9a1828e9f1eb4c30; no voters:
I20260620 14:23:25.195276 22524 leader_election.cc:288] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:23:25.195320 22527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:25.195533 22524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:25.195542 22527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [term 1 LEADER]: Becoming Leader. State: Replica: 21fae5ef37b143ed9a1828e9f1eb4c30, State: Running, Role: LEADER
I20260620 14:23:25.195698 22527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } }
I20260620 14:23:25.196376 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } } }
I20260620 14:23:25.196408 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21fae5ef37b143ed9a1828e9f1eb4c30. Latest consensus state: current_term: 1 leader_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fae5ef37b143ed9a1828e9f1eb4c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39861 } } }
I20260620 14:23:25.196481 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:25.196671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22456
I20260620 14:23:25.196485 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:25.196797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:25.197029 392 external_mini_cluster.cc:1428] Setting key fa9b4d755c9266aa90efef7cbb6c1fe6
W20260620 14:23:25.197533 22543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:25.197616 22543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:25.197669 22538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:25.197839 22538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:25.199496 22538 catalog_manager.cc:1347] Generated new cluster ID: b47f463bcd7744b396e7d9a36c78565d
I20260620 14:23:25.199573 22538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:25.200173 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:25.197670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58567 (local address 127.0.98.62:39861)
0620 14:23:25.197856 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:25.197866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:25.197918 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:23:25.198351 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:25.198359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:25.198395 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:25.198543 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:25.199694 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:25.199699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:25.199699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:25.199704 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:25.199754 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:25.199769 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:25.199823 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:25.199973 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:23:25.200012 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260620 14:23:25.201081 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:25.203815 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:25.202588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59957 (local address 127.0.98.62:39861)
0620 14:23:25.202618 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:25.202620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:25.202893 (+ 273us) server_negotiation.cc:407] Connection header received
0620 14:23:25.202938 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:25.202939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:25.202944 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:25.202997 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:25.203605 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:25.203606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:25.203606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:23:25.203610 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:25.203624 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:23:25.203627 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:25.203678 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:25.203748 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:23:25.203760 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:23:25.219194 22538 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:25.219388 22538 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:25.228430 22538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21fae5ef37b143ed9a1828e9f1eb4c30: Generated new TSK 0
I20260620 14:23:25.228660 22538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:25.233006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22456
2026-06-20T14:23:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1308 ms)
[----------] 18 tests from AuthTokenIssuingTest (23640 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:25 dist-test-slave-h6n5 krb5kdc[22555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:25 dist-test-slave-h6n5 krb5kdc[22555](info): set up 2 sockets
Jun 20 14:23:25 dist-test-slave-h6n5 krb5kdc[22555](info): commencing operation
krb5kdc: starting...
W20260620 14:23:26.271596 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:26.384338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34195,127.0.98.61:39043,127.0.98.60:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:26.462898 22568 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:23:26.463092 22568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:26.463109 22568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:26.463129 22568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:26.464298 22568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:26.464339 22568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:26.464352 22568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:26.464365 22568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:26.464427 22568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:26.465899 22568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34195,127.0.98.61:39043,127.0.98.60:37077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:26.466150 22568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:26.466382 22568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:26.471048 22574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:26.471055 22573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:26.471055 22576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:26.471247 22568 server_base.cc:1034] running on GCE node
I20260620 14:23:26.471472 22568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:26.471804 22568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:26.472956 22568 hybrid_clock.cc:648] HybridClock initialized: now 1781965406472961 us; error 43 us; skew 500 ppm
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:26.475342 22568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:26.475837 22568 webserver.cc:460] Webserver started at http://127.0.98.62:38139/ using document root <none> and password file <none>
I20260620 14:23:26.476071 22568 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:26.476140 22568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:26.476291 22568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:26.477272 22568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "745adbac24184e00b3d180188f43c584"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "b55ad9f91759f6b624a248b51df7f122"
server_key_iv: "a7a8a07a138d012bfaf0bf0e6ee292d3"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.477620 22568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "745adbac24184e00b3d180188f43c584"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "b55ad9f91759f6b624a248b51df7f122"
server_key_iv: "a7a8a07a138d012bfaf0bf0e6ee292d3"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.478916 22568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:23:26.479642 22583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:26.479810 22568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:26.479918 22568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "745adbac24184e00b3d180188f43c584"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "b55ad9f91759f6b624a248b51df7f122"
server_key_iv: "a7a8a07a138d012bfaf0bf0e6ee292d3"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.480003 22568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:26.490940 22568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:26.492898 22568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:26.493055 22568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:26.496994 22568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34195
I20260620 14:23:26.497026 22635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34195 every 8 connection(s)
I20260620 14:23:26.497350 22568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:26.498219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22568
I20260620 14:23:26.498333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:26.498383 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:26.498539 392 external_mini_cluster.cc:1428] Setting key 9f70f3d33d73dc9c0e88629f37dddb08
I20260620 14:23:26.500541 22636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:26.512037 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.501399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.0.98.62:34195)
0620 14:23:26.501517 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.501522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.501552 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:23:26.502055 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.502059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.502081 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.502185 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.502698 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.503506 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.504425 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.504723 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.504863 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.508848 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.508855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.508856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.508864 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.510425 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.510892 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.510894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.510894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.511021 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.511280 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.511281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.511281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.511470 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.511531 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.511787 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.511892 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
I20260620 14:23:26.512179 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.501217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34195 (local address 127.0.0.1:36492)
0620 14:23:26.501373 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.501405 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.501900 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.502193 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.502195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.502457 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.502600 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.502604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.503678 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.503679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.504326 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.504328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.504844 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.504845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.504908 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.505453 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.505461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.508758 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.510512 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.510519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.510521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.510810 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.511096 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.511097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.511098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.511212 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.511543 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.511575 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.511654 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.511767 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.512000 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":63,"thread_start_us":77,"threads_started":1}
I20260620 14:23:26.512974 22636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:26.513325 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.513198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44856)
0620 14:23:26.513253 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.513302 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:23:26.513437 22584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
I20260620 14:23:26.513847 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.513760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44860)
0620 14:23:26.513803 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.513829 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:23:26.513967 22636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39043: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:23:26.522500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39043
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34195,127.0.98.61:39043,127.0.98.60:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:26.563187 22636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } attempt: 1
I20260620 14:23:26.563736 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.563601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44874)
0620 14:23:26.563666 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.563708 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:23:26.564136 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.564033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44888)
0620 14:23:26.564079 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.564106 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:23:26.564261 22636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39043: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
W20260620 14:23:26.600569 22643 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:23:26.600800 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:26.600831 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:26.600873 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:26.601991 22643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:26.602056 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:26.602082 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:26.602111 22643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:26.602135 22643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:26.603600 22643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34195,127.0.98.61:39043,127.0.98.60:37077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:26.603866 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:26.604092 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:26.608543 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:26.608548 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:26.608577 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:26.608819 22643 server_base.cc:1034] running on GCE node
I20260620 14:23:26.609030 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:26.609249 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:26.610388 22643 hybrid_clock.cc:648] HybridClock initialized: now 1781965406610377 us; error 27 us; skew 500 ppm
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:26.612847 22643 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:23:26.613200 22643 webserver.cc:460] Webserver started at http://127.0.98.61:45677/ using document root <none> and password file <none>
I20260620 14:23:26.613365 22643 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:26.613404 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:26.613488 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:26.614439 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data/instance:
uuid: "e224d9db6a764c20b1d1b46f0d893267"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "c5cd241ce63eb4baa6e62c9fbce4f18d"
server_key_iv: "9144a700cd6a487bd6c63a0b25815ec5"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.614720 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/wal/instance:
uuid: "e224d9db6a764c20b1d1b46f0d893267"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "c5cd241ce63eb4baa6e62c9fbce4f18d"
server_key_iv: "9144a700cd6a487bd6c63a0b25815ec5"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.615988 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:26.616627 22658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:26.616837 22643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:23:26.616928 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/wal
uuid: "e224d9db6a764c20b1d1b46f0d893267"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "c5cd241ce63eb4baa6e62c9fbce4f18d"
server_key_iv: "9144a700cd6a487bd6c63a0b25815ec5"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.617008 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:26.632405 22636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } attempt: 2
I20260620 14:23:26.632808 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.632671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44898)
0620 14:23:26.632730 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.632773 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:23:26.633159 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.633079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44900)
0620 14:23:26.633119 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.633143 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:23:26.633246 22636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39043: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39043: connect: Connection refused (error 111)
I20260620 14:23:26.642823 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:26.644754 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:26.644941 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:26.648957 22643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39043
I20260620 14:23:26.649024 22710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39043 every 8 connection(s)
I20260620 14:23:26.649367 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:23:26.650359 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:26.651953 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:26.656827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22643
I20260620 14:23:26.656939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-1/wal/instance
I20260620 14:23:26.657130 392 external_mini_cluster.cc:1428] Setting key efe70e36cc149e908ccc06b596cedba7
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:26.661152 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.652614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.0.98.62:34195)
0620 14:23:26.652673 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.652675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.652883 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:23:26.653319 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.653320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.653331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.653389 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.653959 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.654423 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.655335 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.655517 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.655612 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.658099 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.658100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.658100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.658107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.659699 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.660083 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.660085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.660085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.660155 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.660383 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.660384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.660384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.660541 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.660617 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.660941 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.661035 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:23:26.661175 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.652588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34195 (local address 127.0.0.1:36494)
0620 14:23:26.652758 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.652799 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.653233 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.653451 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.653455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.653696 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.653890 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.653894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.654494 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.654495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.655267 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.655269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.655641 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.655642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.655747 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.656262 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.656270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.657989 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.659789 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.659795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.659797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.660010 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.660222 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.660222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.660222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.660317 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.660608 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.660612 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.660801 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.660887 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.661009 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260620 14:23:26.661849 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:23:26.671826 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.662123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44904)
0620 14:23:26.662350 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.662363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.662420 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.662559 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.662561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.662731 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.662793 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.662795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.663649 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.663651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.664493 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.664496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.664953 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.664954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.665031 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.665424 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.665429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.668625 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.670560 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.670561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.670562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.670795 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.671003 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.671004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.671004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.671075 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.671340 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.671341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.671390 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.671476 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.671618 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260620 14:23:26.671830 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.662199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.98.61:39043)
0620 14:23:26.662360 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.662363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.662412 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:23:26.662480 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.662482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.662493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.662552 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.662879 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.663537 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.664581 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.664880 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.664929 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.668717 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.668726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.668727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.668735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.670476 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.670885 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.670887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.670887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.670937 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.671134 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.671135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.671135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.671272 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.671331 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.671527 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.671696 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260620 14:23:26.672667 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:26.673064 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.672922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59606)
0620 14:23:26.672977 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.673035 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:23:26.673246 22660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
I20260620 14:23:26.673677 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.673563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59612)
0620 14:23:26.673620 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.673650 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:23:26.673796 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:23:26.682116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37077
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34195,127.0.98.61:39043,127.0.98.60:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:26.723059 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } attempt: 1
I20260620 14:23:26.723605 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.723441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59616)
0620 14:23:26.723508 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.723569 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:23:26.723987 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.723897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59630)
0620 14:23:26.723943 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.723969 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:23:26.724083 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
I20260620 14:23:26.724376 22636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } attempt: 3
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:23:26.731446 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.724630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44916)
0620 14:23:26.724688 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.724700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.724754 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.724912 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.724913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.725057 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.725122 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.725124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.726066 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.726067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.726582 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.726584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.727062 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.727063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.727119 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.727632 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.727636 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.728820 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.730506 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.730507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.730507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.730674 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.730882 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.730882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.730883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.730941 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.731196 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.731197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.731244 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.731299 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.731376 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:23:26.731510 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.724707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.0.98.61:39043)
0620 14:23:26.724759 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.724761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.724775 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:23:26.724812 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.724813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.724823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.724901 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.725192 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.725962 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.726661 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.726985 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.727059 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.728899 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.728901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.728902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.728909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.730423 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.730745 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.730746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.730746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.730816 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.731000 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.731001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.731001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.731133 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.731206 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.731341 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.731423 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:23:26.732033 22636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:26.732388 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.732275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59638)
0620 14:23:26.732324 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.732358 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:23:26.732700 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.732613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59650)
0620 14:23:26.732656 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.732682 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:23:26.732805 22636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
I20260620 14:23:26.763953 22636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } attempt: 1
I20260620 14:23:26.764478 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.764314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59662)
0620 14:23:26.764388 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.764440 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:23:26.764881 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.764766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59668)
0620 14:23:26.764824 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.764856 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:23:26.765021 22636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
W20260620 14:23:26.765684 22718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:23:26.765913 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:26.765950 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:26.765992 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:26.767131 22718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:26.767196 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:26.767222 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:26.767251 22718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:26.767277 22718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:26.768827 22718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34195,127.0.98.61:39043,127.0.98.60:37077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:26.769097 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:26.769327 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:26.774495 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:26.774474 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:26.774477 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:26.778152 22718 server_base.cc:1034] running on GCE node
I20260620 14:23:26.778451 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:26.778837 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:26.779981 22718 hybrid_clock.cc:648] HybridClock initialized: now 1781965406779963 us; error 41 us; skew 500 ppm
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:26.783105 22718 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:23:26.783627 22718 webserver.cc:460] Webserver started at http://127.0.98.60:45621/ using document root <none> and password file <none>
I20260620 14:23:26.783874 22718 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:26.783944 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:26.784081 22718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:26.785144 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data/instance:
uuid: "31ad2505f9a84983b72166496620f267"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "96a5a7840c5eedbe31e73ff1e5241774"
server_key_iv: "dba724419f97eb14c109705ff26b312c"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.785516 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/wal/instance:
uuid: "31ad2505f9a84983b72166496620f267"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "96a5a7840c5eedbe31e73ff1e5241774"
server_key_iv: "dba724419f97eb14c109705ff26b312c"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.786943 22718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:26.787768 22733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:26.787946 22718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:23:26.788044 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/wal
uuid: "31ad2505f9a84983b72166496620f267"
format_stamp: "Formatted at 2026-06-20 14:23:26 on dist-test-slave-h6n5"
server_key: "96a5a7840c5eedbe31e73ff1e5241774"
server_key_iv: "dba724419f97eb14c109705ff26b312c"
server_key_version: "encryptionkey@0"
I20260620 14:23:26.788125 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:26.792223 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } attempt: 2
I20260620 14:23:26.792654 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.792529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59682)
0620 14:23:26.792585 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.792622 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:23:26.793047 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.792967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59686)
0620 14:23:26.793008 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.793031 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:23:26.793171 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37077: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37077: connect: Connection refused (error 111)
I20260620 14:23:26.799347 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:26.801316 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:26.801465 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:26.805732 22718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37077
I20260620 14:23:26.805931 22785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37077 every 8 connection(s)
I20260620 14:23:26.806161 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:23:26.806866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22718
I20260620 14:23:26.806981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965139717380-392-0/minicluster-data/master-2/wal/instance
I20260620 14:23:26.807168 392 external_mini_cluster.cc:1428] Setting key bc8f8dae2674c7941bcd15dbcf0e3d5e
I20260620 14:23:26.807178 22786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:26.809733 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:26.818521 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.807736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.0.98.60:37077)
0620 14:23:26.807914 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.807924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.807960 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:23:26.808367 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.808374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.808419 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.808532 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.809381 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.810443 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.811475 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.812096 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.812170 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.814952 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.814958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.814958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.814981 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.816870 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.817263 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.817265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.817265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.817371 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.817567 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.817568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.817568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.817869 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.818007 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.818163 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.818295 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260620 14:23:26.818594 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.810303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34195 (local address 127.0.0.1:36508)
0620 14:23:26.810447 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.810495 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.810573 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.810709 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.810712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.811067 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.811146 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.811148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.811977 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.811977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.812775 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.812779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.813140 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.813142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.813238 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.813717 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.813728 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.815517 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.817366 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.817371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.817373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.817644 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.817829 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.817830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.817831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.817896 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.818176 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.818181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.818257 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.818378 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.818470 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260620 14:23:26.818622 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.810391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.0.98.62:34195)
0620 14:23:26.810456 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.810458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.810576 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:23:26.810638 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.810639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.810649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.810690 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.811220 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.811880 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.812862 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.813038 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.813092 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.815644 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.815645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.815646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.815652 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.817299 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.817715 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.817716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.817716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.817765 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.817957 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.817958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.817959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.818110 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.818171 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.818433 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.818511 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:23:26.819195 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:26.826999 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.819439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39043 (local address 127.0.0.1:44932)
0620 14:23:26.819530 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.819539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.819602 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.819717 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.819717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.819868 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.819927 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.819929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.820787 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.820788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.821528 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.821530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.822176 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.822178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.822261 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.822805 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.822810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.824209 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.825847 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.825848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.825848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.826059 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.826375 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.826375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.826376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.826425 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.826701 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.826702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.826739 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.826789 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.826857 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:23:26.827008 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.819498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.98.61:39043)
0620 14:23:26.819544 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.819565 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.819598 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:23:26.819654 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.819655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.819666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.819713 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.820006 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.820709 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.821627 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.821907 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.821979 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.824302 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.824303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.824304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.824312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.825786 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.826241 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.826242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.826242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.826306 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.826489 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.826490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.826490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.826633 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.826710 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.826836 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.826919 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:23:26.827579 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:26.828317 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.819339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.98.62:34195)
0620 14:23:26.819380 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.819382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.819529 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:23:26.819572 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.819573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.819583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.819639 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.820230 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.820886 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.821810 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.822077 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.822125 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.825005 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.825006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.825006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.825013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.826839 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.827434 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.827435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.827436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.827497 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.827784 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.827786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.827786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.827927 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.828113 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.828157 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.828217 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:23:26.834719 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.827931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.0.98.60:37077)
0620 14:23:26.827965 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.827967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.827974 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:23:26.827996 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.827996 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.828006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.828055 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.828343 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.828917 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.829797 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.830048 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.830097 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.831826 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.831827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.831827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.831835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.833360 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.833741 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.833742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.833743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.833801 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.834007 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.834008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.834009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.834143 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.834192 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.834472 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.834583 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":17}
I20260620 14:23:26.834719 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.827857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59702)
0620 14:23:26.827908 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.827919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.827954 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.828058 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.828059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.828203 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.828254 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.828256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.829010 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.829011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.829700 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.829702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.830116 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.830117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.830172 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.830541 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.830546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.831742 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.833440 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.833441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.833441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.833668 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.833888 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.833889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.833889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.833942 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.834309 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.834310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.834357 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.834431 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.834517 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:23:26.836683 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: Bootstrap starting.
I20260620 14:23:26.837318 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.829245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.0.98.61:39043)
0620 14:23:26.829321 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.829323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.829331 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:23:26.829350 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.829351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.829360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.829421 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.829866 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.830451 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.831123 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.831337 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.831408 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.834071 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.834072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.834072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.834078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.836100 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.836450 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.836451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.836451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.836512 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.836694 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.836695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.836695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.836831 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.836981 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.837185 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.837246 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:23:26.837749 22786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:26.838028 22786 log.cc:824] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:26.838907 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:26.839051 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: No bootstrap required, opened a new log
I20260620 14:23:26.840180 22636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } attempt: 2
I20260620 14:23:26.839896 22610 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "745adbac24184e00b3d180188f43c584"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36518
I20260620 14:23:26.841210 22786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.841377 22786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ad2505f9a84983b72166496620f267, State: Initialized, Role: FOLLOWER
I20260620 14:23:26.841454 22786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.841784 22786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:26.841727 22792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.841892 22792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:23:26.843741 22803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:26.843796 22803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:23:26.843998 22718 master_runner.cc:186] Error getting master registration for 127.0.98.62:34195: OK, instance_id { permanent_uuid: "745adbac24184e00b3d180188f43c584" instance_seqno: 1781965406495882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:23:26.846807 22718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:23:26.848197 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.840458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.0.98.60:37077)
0620 14:23:26.840574 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.840576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.840583 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:23:26.840600 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.840600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.840609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.840659 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.841008 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.841688 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.842717 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.843050 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.843121 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.845311 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.845313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.845313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.845320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.846986 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.847386 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.847387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.847387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.847446 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.847651 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.847652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.847652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.847807 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.847885 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.848038 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.848114 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 14:23:26.848196 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.840415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59708)
0620 14:23:26.840460 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.840471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.840507 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.840685 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.840686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.840852 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.840926 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.840928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.841800 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.841801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.842618 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.842621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.843119 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.843120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.843201 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.843755 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.843760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.845205 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.847074 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.847075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.847076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.847301 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.847515 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.847516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.847516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.847601 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.847878 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.847879 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.847927 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.847996 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.848094 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:23:26.849761 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584: Bootstrap starting.
I20260620 14:23:26.850374 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:26.850673 22636 log.cc:824] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:26.851437 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584: No bootstrap required, opened a new log
I20260620 14:23:26.852908 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.853052 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745adbac24184e00b3d180188f43c584, State: Initialized, Role: FOLLOWER
I20260620 14:23:26.853138 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.853405 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.853475 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:26.853485 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:26.853567 22610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:23:26.853642 22610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:26.854460 22610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.854691 22610 leader_election.cc:288] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [CANDIDATE]: Term 1 election: Requested vote from peers e224d9db6a764c20b1d1b46f0d893267 (127.0.98.61:39043), 31ad2505f9a84983b72166496620f267 (127.0.98.60:37077)
W20260620 14:23:26.855237 22817 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:26.855286 22817 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:26.855851 22760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "745adbac24184e00b3d180188f43c584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31ad2505f9a84983b72166496620f267"
I20260620 14:23:26.855973 22760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:26.856798 22760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745adbac24184e00b3d180188f43c584 in term 1.
I20260620 14:23:26.857115 22585 leader_election.cc:302] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ad2505f9a84983b72166496620f267, 745adbac24184e00b3d180188f43c584; no voters:
I20260620 14:23:26.857249 22806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:26.857383 22806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [term 1 LEADER]: Becoming Leader. State: Replica: 745adbac24184e00b3d180188f43c584, State: Running, Role: LEADER
I20260620 14:23:26.857455 22806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.857847 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745adbac24184e00b3d180188f43c584. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.857928 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:26.856204 22568 master_runner.cc:186] Error getting master registration for 127.0.98.61:39043: OK, instance_id { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" instance_seqno: 1781965406647864 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:23:26.858966 22568 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:23:26.860114 22821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:26.860335 22821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:26.862318 22760 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [term 1 FOLLOWER]: Refusing update from remote peer 745adbac24184e00b3d180188f43c584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:26.862614 22806 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:26.863902 22792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745adbac24184e00b3d180188f43c584. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.863992 22792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:26.865130 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.865216 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:26.865607 22792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.865686 22792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:26.865935 22826 mvcc.cc:205] Tried to move back new op lower bound from 7298930306504224768 to 7298930306488995840. Current Snapshot: MvccSnapshot[applied={T|T < 7298930306504224768}]
I20260620 14:23:26.866641 22830 mvcc.cc:205] Tried to move back new op lower bound from 7298930306504224768 to 7298930306488995840. Current Snapshot: MvccSnapshot[applied={T|T < 7298930306504224768}]
I20260620 14:23:26.871837 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.861195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.0.98.61:39043)
0620 14:23:26.861243 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.861245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.861979 (+ 734us) server_negotiation.cc:407] Connection header received
0620 14:23:26.862102 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.862103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.862119 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.862157 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.863152 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.863882 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.865032 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.865346 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.865390 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.867283 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.867285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.867285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.867293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.869132 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.869708 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.869709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.869710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.869784 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.870815 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.870816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.870816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.870967 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.871534 (+ 567us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.871658 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.871739 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:23:26.876757 22821 catalog_manager.cc:1347] Generated new cluster ID: 2e12eef2accb4d089a58ae887b696a43
I20260620 14:23:26.876817 22821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:26.877030 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.861641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.0.98.60:37077)
0620 14:23:26.861686 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.861688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.867839 (+ 6151us) server_negotiation.cc:407] Connection header received
0620 14:23:26.867927 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.867928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.867937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.867965 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.868590 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.869337 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.870828 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.871135 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.871175 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.873472 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.873474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.873474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.873481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.875194 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.875862 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.875864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.875864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.875941 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.876296 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.876298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.876298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.876451 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.876551 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.876866 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.876943 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:23:26.884591 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } attempt: 3
I20260620 14:23:26.885038 22821 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:26.886078 22821 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 14:23:26 dist-test-slave-h6n5 krb5kdc[22555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:23:26.892412 22821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584: Generated new TSK 0
I20260620 14:23:26.892596 22821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:23:26.893195 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.884867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37077 (local address 127.0.0.1:59726)
0620 14:23:26.884918 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:26.884929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:26.884968 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:26.885110 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:26.885111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:26.885327 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:26.885374 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.885376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.886199 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.886201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.887015 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:26.887018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.887456 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.887457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:26.887514 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.888005 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:26.888010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:26.889480 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:26.891655 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.891656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.891656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.891887 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.892182 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:26.892183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:26.892183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:26.892262 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:26.892842 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:26.892843 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:26.892905 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 14:23:26.892970 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:23:26.893075 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:23:26.893944 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.884897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.0.98.60:37077)
0620 14:23:26.884939 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.884941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.884960 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:23:26.885046 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.885047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.885057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.885104 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.885454 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.886120 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.887101 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.887399 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.887454 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.889560 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.889562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.889562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.889568 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.891489 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.891987 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.891988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.891989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.892075 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.892348 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.892349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.892350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.892711 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.893621 (+ 910us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.893689 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.893809 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:23:26.895686 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: Bootstrap starting.
I20260620 14:23:26.895763 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:26.861409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.0.98.62:34195)
0620 14:23:26.861450 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.861452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.867685 (+ 6233us) server_negotiation.cc:407] Connection header received
0620 14:23:26.867752 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.867753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.867763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.867825 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.876585 (+ 8760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.877330 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.884196 (+ 6866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:26.884493 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:26.884540 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:26.892426 (+ 7886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:26.892428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:26.892428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:26.892435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:26.894121 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.894708 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.894709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.894710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.894779 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:26.895045 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:26.895046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:26.895046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:26.895166 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:26.895256 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:26.895579 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:23:26.895677 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:23:26.896420 22711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:26.896696 22711 log.cc:824] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:26.900424 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: No bootstrap required, opened a new log
I20260620 14:23:26.902506 22711 raft_consensus.cc:348] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.902642 22711 raft_consensus.cc:729] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e224d9db6a764c20b1d1b46f0d893267, State: Initialized, Role: FOLLOWER
I20260620 14:23:26.902717 22711 consensus_queue.cc:260] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } }
I20260620 14:23:26.903015 22711 sys_catalog.cc:564] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:26.903748 22684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:26.904646 22684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [term 1 FOLLOWER]: Refusing update from remote peer 745adbac24184e00b3d180188f43c584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:26.904973 22806 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:26.905021 22685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "745adbac24184e00b3d180188f43c584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e224d9db6a764c20b1d1b46f0d893267"
I20260620 14:23:26.905757 22685 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745adbac24184e00b3d180188f43c584 in term 1.
I20260620 14:23:26.907191 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745adbac24184e00b3d180188f43c584. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.907311 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:26.907370 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.907413 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:26.907608 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:26.907752 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:26.908370 22845 mvcc.cc:205] Tried to move back new op lower bound from 7298930306624978944 to 7298930306488995840. Current Snapshot: MvccSnapshot[applied={T|T < 7298930306504224768}]
W20260620 14:23:26.908900 22852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:26.908993 22852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 14:23:26.925434 22640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.923640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.98.62:34195)
0620 14:23:26.923693 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.923696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.924758 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 14:23:26.924860 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.924861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.924873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.924904 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.925364 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36526: BlockingRecv error: recv got EOF from 127.0.0.1:36526 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:23:26.926384 22789 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.923991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.0.98.60:37077)
0620 14:23:26.924029 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.924031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.925660 (+ 1629us) server_negotiation.cc:407] Connection header received
0620 14:23:26.925756 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.925757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.925767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.925796 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.926325 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59738: BlockingRecv error: recv got EOF from 127.0.0.1:59738 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:23:26.927069 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.924287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.0.98.61:39043)
0620 14:23:26.924323 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.924325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.926593 (+ 2268us) server_negotiation.cc:407] Connection header received
0620 14:23:26.926688 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.926689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.926700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.926729 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.927000 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44952: BlockingRecv error: recv got EOF from 127.0.0.1:44952 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:23:26.937885 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.937122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.98.61:39043)
0620 14:23:26.937167 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.937169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.937361 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:23:26.937462 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.937463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.937483 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.937512 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.937836 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44968: BlockingRecv error: recv got EOF from 127.0.0.1:44968 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:23:26.944303 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.940082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.0.98.61:39043)
0620 14:23:26.940113 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.940115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.943790 (+ 3675us) server_negotiation.cc:407] Connection header received
0620 14:23:26.943889 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.943890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.943900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.943929 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.944253 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44976: BlockingRecv error: recv got EOF from 127.0.0.1:44976 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:23:26.948173 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.947333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.98.61:39043)
0620 14:23:26.947365 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.947367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.947590 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:23:26.947684 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.947685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.947694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.947723 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.948128 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44986: BlockingRecv error: recv got EOF from 127.0.0.1:44986 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:23:26.954443 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.953599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.98.61:39043)
0620 14:23:26.953651 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.953653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.953903 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:23:26.953998 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.953998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.954009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.954038 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.954368 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44988: BlockingRecv error: recv got EOF from 127.0.0.1:44988 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 14:23:26.964571 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.963751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.0.98.61:39043)
0620 14:23:26.963789 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.963791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.964011 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:23:26.964111 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.964112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.964123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.964153 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.964508 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45004: BlockingRecv error: recv got EOF from 127.0.0.1:45004 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:23:26.983108 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:26.982068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.0.98.61:39043)
0620 14:23:26.982122 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:26.982124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:26.982450 (+ 326us) server_negotiation.cc:407] Connection header received
0620 14:23:26.982563 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:26.982564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:26.982578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:26.982627 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:26.983050 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45012: BlockingRecv error: recv got EOF from 127.0.0.1:45012 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:23:27.017517 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:27.016560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.0.98.61:39043)
0620 14:23:27.016608 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.016610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.016886 (+ 276us) server_negotiation.cc:407] Connection header received
0620 14:23:27.016992 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.016993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.017005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.017036 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:27.017455 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45020: BlockingRecv error: recv got EOF from 127.0.0.1:45020 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:23:27.083791 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:27.082921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.98.61:39043)
0620 14:23:27.082999 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.083001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.083138 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:23:27.083214 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.083215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.083227 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.083322 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:27.083726 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45026: BlockingRecv error: recv got EOF from 127.0.0.1:45026 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:23:27.213737 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:27.213018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.0.98.61:39043)
0620 14:23:27.213089 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.213092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.213175 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:23:27.213216 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.213217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.213228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.213284 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:27.213647 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45028: BlockingRecv error: recv got EOF from 127.0.0.1:45028 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:23:27.311080 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "745adbac24184e00b3d180188f43c584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745adbac24184e00b3d180188f43c584" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34195 } } peers { permanent_uuid: "e224d9db6a764c20b1d1b46f0d893267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39043 } } peers { permanent_uuid: "31ad2505f9a84983b72166496620f267" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37077 } } }
I20260620 14:23:27.311295 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745adbac24184e00b3d180188f43c584 [sys.catalog]: This master's current role is: LEADER
W20260620 14:23:27.471788 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:27.471005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.0.98.61:39043)
0620 14:23:27.471066 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.471070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.471131 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:23:27.471179 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.471179 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.471189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.471264 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:27.471707 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45036: BlockingRecv error: recv got EOF from 127.0.0.1:45036 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:23:27.844254 22803 catalog_manager.cc:1259] Loaded cluster ID: 2e12eef2accb4d089a58ae887b696a43
I20260620 14:23:27.844336 22803 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: loading cluster ID for follower catalog manager: success
I20260620 14:23:27.845119 22803 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: acquiring CA information for follower catalog manager: success
I20260620 14:23:27.845373 22803 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31ad2505f9a84983b72166496620f267: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:23:27.909463 22852 catalog_manager.cc:1259] Loaded cluster ID: 2e12eef2accb4d089a58ae887b696a43
I20260620 14:23:27.909577 22852 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: loading cluster ID for follower catalog manager: success
I20260620 14:23:27.910521 22852 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: acquiring CA information for follower catalog manager: success
I20260620 14:23:27.910840 22852 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e224d9db6a764c20b1d1b46f0d893267: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:23:27.989180 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:27.985127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.0.98.61:39043)
0620 14:23:27.985243 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.985245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.985339 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:23:27.985420 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.985421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.985499 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.985606 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:27.985806 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.986583 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.987930 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.988301 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.988371 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:27.988589 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:27.988745 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:27.988826 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:27.989037 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:23:27.989060 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260620 14:23:27.994567 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:27.991234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.0.98.60:37077)
0620 14:23:27.991339 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.991342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.991628 (+ 286us) server_negotiation.cc:407] Connection header received
0620 14:23:27.991800 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.991802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.991850 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.991931 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:27.992111 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.992796 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.993659 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.993959 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.994031 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:27.994188 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:27.994324 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:27.994413 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:27.994429 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 14:23:27.994452 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260620 14:23:27.999361 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:27.996357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.0.98.62:34195)
0620 14:23:27.996478 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.996481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.996489 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:23:27.996535 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.996536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.996585 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.996665 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:27.996833 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.997448 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.998415 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.998681 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.998761 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:27.998938 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:27.999052 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:27.999108 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:27.999265 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:23:27.999288 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260620 14:23:27.999343 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:27.996476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.98.61:39043)
0620 14:23:27.996518 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.996521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.996634 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:23:27.996720 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.996721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.996731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.996789 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:27.996933 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.997658 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.998532 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.998847 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.998911 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:27.999083 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:27.999116 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:27.999164 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:27.999243 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:23:27.999269 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:23:27.999709 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:27.996588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.98.60:37077)
0620 14:23:27.996635 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:27.996637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:27.996815 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:23:27.996898 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:27.996899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:27.996907 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:27.996954 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:27.997119 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.997807 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.998823 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:27.999097 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:27.999168 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:27.999394 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:27.999447 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:27.999482 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:27.999624 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:23:27.999645 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:23:28.001201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22568
I20260620 14:23:28.007217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22643
I20260620 14:23:28.012367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22718
2026-06-20T14:23:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2780 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:23:28 dist-test-slave-h6n5 krb5kdc[22938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:23:28 dist-test-slave-h6n5 krb5kdc[22938](info): set up 2 sockets
Jun 20 14:23:28 dist-test-slave-h6n5 krb5kdc[22938](info): commencing operation
krb5kdc: starting...
W20260620 14:23:29.043871 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/test-user.keytab.
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:23:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:23:29.158571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35629,127.0.98.61:43427,127.0.98.60:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:23:29.239500 22951 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:23:29.239820 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:29.239851 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:29.239894 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:29.241034 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:29.241101 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:29.241125 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:29.241146 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:29.241175 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:29.242702 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35629,127.0.98.61:43427,127.0.98.60:42119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:29.242991 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:29.243232 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:29.248179 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:29.248175 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:29.248170 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:29.248205 22951 server_base.cc:1034] running on GCE node
I20260620 14:23:29.248610 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:29.248921 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:29.250118 22951 hybrid_clock.cc:648] HybridClock initialized: now 1781965409250104 us; error 42 us; skew 500 ppm
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:29.253167 22951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:23:29.253638 22951 webserver.cc:460] Webserver started at http://127.0.98.62:44735/ using document root <none> and password file <none>
I20260620 14:23:29.253892 22951 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:29.253938 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:29.254077 22951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:29.255146 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data/instance:
uuid: "a03dd28350f34f20a67c73740ba5e037"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "8c017d88d3b22d7fe6b8166c6e1cae49"
server_key_iv: "48bec28a6ed4e9277be8e7f660fd6c19"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.255496 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/wal/instance:
uuid: "a03dd28350f34f20a67c73740ba5e037"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "8c017d88d3b22d7fe6b8166c6e1cae49"
server_key_iv: "48bec28a6ed4e9277be8e7f660fd6c19"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.257205 22951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:29.258122 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:29.258294 22951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:29.258389 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/wal
uuid: "a03dd28350f34f20a67c73740ba5e037"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "8c017d88d3b22d7fe6b8166c6e1cae49"
server_key_iv: "48bec28a6ed4e9277be8e7f660fd6c19"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.258462 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:29.286181 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:29.288384 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:29.288554 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:29.292455 22951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35629
I20260620 14:23:29.292505 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35629 every 8 connection(s)
I20260620 14:23:29.292953 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:23:29.293467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 22951
I20260620 14:23:29.293578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-0/wal/instance
I20260620 14:23:29.293824 392 external_mini_cluster.cc:1428] Setting key a62b57a2f9980755cc923c4644368463
I20260620 14:23:29.294327 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:29.296602 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:29.309224 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.297389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.0.98.62:35629)
0620 14:23:29.298202 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.298206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.298216 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:23:29.298274 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.298276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.298290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.298353 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.299202 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.300139 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.301501 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.301865 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.301951 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.305656 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.305665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.305665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.305673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.307415 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.307911 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.307914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.307914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.308014 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.308324 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.308325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.308325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.308577 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.308646 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.308914 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.309055 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":794,"thread_start_us":85,"threads_started":1}
I20260620 14:23:29.309345 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.297352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35629 (local address 127.0.0.1:55570)
0620 14:23:29.297596 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.297629 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.298074 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.298419 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.298422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.298817 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.299064 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.299071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.300241 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.300242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.301394 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.301397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.301974 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.301976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.302060 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.302539 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.302547 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.305568 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.307511 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.307517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.307520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.307831 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.308127 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.308128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.308129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.308250 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.308657 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.308664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.308749 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.308862 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.309150 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":77,"thread_start_us":73,"threads_started":1}
I20260620 14:23:29.310161 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:29.310555 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.310419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34378)
0620 14:23:29.310468 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.310524 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:23:29.310707 22968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
I20260620 14:23:29.311151 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.311040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34386)
0620 14:23:29.311092 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.311123 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:23:29.311288 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43427: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:23:29.319767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43427
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35629,127.0.98.61:43427,127.0.98.60:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:29.360541 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } attempt: 1
I20260620 14:23:29.361120 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.360954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34392)
0620 14:23:29.361025 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.361080 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:23:29.361588 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.361490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34396)
0620 14:23:29.361533 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.361563 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:23:29.361742 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43427: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
W20260620 14:23:29.400717 23026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:23:29.400928 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:29.400945 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:29.400965 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:29.402145 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:29.402187 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:29.402201 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:29.402215 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:29.402226 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:29.403748 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35629,127.0.98.61:43427,127.0.98.60:42119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:29.404016 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:29.404256 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:29.409157 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:29.409152 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:29.409188 23026 server_base.cc:1034] running on GCE node
W20260620 14:23:29.409129 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:29.409680 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:29.410022 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:29.411195 23026 hybrid_clock.cc:648] HybridClock initialized: now 1781965409411181 us; error 29 us; skew 500 ppm
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:29.413736 23026 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:23:29.414273 23026 webserver.cc:460] Webserver started at http://127.0.98.61:36267/ using document root <none> and password file <none>
I20260620 14:23:29.414511 23026 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:29.414579 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:29.414728 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:29.415884 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data/instance:
uuid: "7d014893637d4d819854a01499d1c369"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "327527721a898173e8a7d922e5001738"
server_key_iv: "77e6aff9dbe82da708a2c7294d9ff1cd"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.416247 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/wal/instance:
uuid: "7d014893637d4d819854a01499d1c369"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "327527721a898173e8a7d922e5001738"
server_key_iv: "77e6aff9dbe82da708a2c7294d9ff1cd"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.417557 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:23:29.418309 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:29.418475 23026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:23:29.418581 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/wal
uuid: "7d014893637d4d819854a01499d1c369"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "327527721a898173e8a7d922e5001738"
server_key_iv: "77e6aff9dbe82da708a2c7294d9ff1cd"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.418654 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:29.429910 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } attempt: 2
I20260620 14:23:29.430423 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.430272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34410)
0620 14:23:29.430335 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.430379 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:23:29.430823 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.430691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34422)
0620 14:23:29.430759 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.430798 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:23:29.430970 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43427: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43427: connect: Connection refused (error 111)
I20260620 14:23:29.435984 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:29.438206 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:29.438387 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:29.442556 23026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43427
I20260620 14:23:29.442638 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43427 every 8 connection(s)
I20260620 14:23:29.442981 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:23:29.443970 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:29.444577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 23026
I20260620 14:23:29.444696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-1/wal/instance
I20260620 14:23:29.444892 392 external_mini_cluster.cc:1428] Setting key 185f0d5830a3ab59c28df308cf2a3d12
I20260620 14:23:29.445703 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 14:23:29.455842 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.446415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.0.98.62:35629)
0620 14:23:29.446492 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.446494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.446747 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:23:29.447389 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.447390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.447405 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.447468 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.448217 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.449072 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.450140 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.450500 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.450580 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.452872 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.452873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.452873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.452880 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.454352 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.454789 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.454790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.454790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.454856 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.455148 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.455150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.455150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.455275 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.455355 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.455660 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.455747 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:23:29.456032 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.446389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35629 (local address 127.0.0.1:55586)
0620 14:23:29.446598 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.446652 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.447266 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.447649 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.447654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.447908 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.448115 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.448119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.449184 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.449185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.450042 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.450044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.450608 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.450609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.450726 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.451294 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.451303 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.452777 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.454426 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.454432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.454433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.454718 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.454955 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.454955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.454956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.455057 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.455345 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.455352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.455488 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.455637 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.455799 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260620 14:23:29.456605 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:23:29.464844 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.456923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.98.61:43427)
0620 14:23:29.457109 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.457116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.457139 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:23:29.457163 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.457165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.457179 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.457261 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.457504 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.458658 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.459329 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.459903 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.459988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.461757 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.461766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.461766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.461773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.463467 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.463910 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.463914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.463914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.463971 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.464177 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.464178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.464179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.464348 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.464439 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.464554 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.464709 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260620 14:23:29.465155 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.456868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34436)
0620 14:23:29.456915 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.456926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.456965 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.457275 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.457276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.457385 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.457430 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.457431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.458748 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.458749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.459240 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.459242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.459982 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.459983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.460037 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.460517 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.460521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.461692 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.463649 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.463650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.463651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.463816 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.464045 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.464046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.464046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.464099 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.464427 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.464428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.464465 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.464535 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.465070 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":437}
I20260620 14:23:29.465763 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:29.466126 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.465988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52530)
0620 14:23:29.466047 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.466097 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:23:29.466251 23043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
I20260620 14:23:29.466645 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.466554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52534)
0620 14:23:29.466601 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.466627 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:23:29.466743 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42119: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:23:29.470347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTTnugX/build/release/bin/kudu
/tmp/dist-test-taskTTnugX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42119
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35629,127.0.98.61:43427,127.0.98.60:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:23:29.515969 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } attempt: 1
I20260620 14:23:29.516462 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.516319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52542)
0620 14:23:29.516388 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.516429 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:23:29.516815 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.516718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52544)
0620 14:23:29.516764 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.516788 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:23:29.516901 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42119: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
I20260620 14:23:29.522121 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } attempt: 3
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:23:29.529593 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.522528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.0.98.61:43427)
0620 14:23:29.522600 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.522602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.522613 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:23:29.522688 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.522689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.522697 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.522745 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.523139 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.523713 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.524714 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.524925 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.524975 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.526940 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.526942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.526942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.526949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.528424 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.528802 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.528803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.528804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.528864 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.529076 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.529077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.529078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.529190 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.529263 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.529424 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.529498 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:23:29.529603 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.522469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34446)
0620 14:23:29.522542 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.522557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.522622 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.522771 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.522772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.522970 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.523064 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.523066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.523789 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.523790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.524618 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.524622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.524994 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.524995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.525079 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.525515 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.525521 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.526834 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.528529 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.528530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.528531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.528715 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.528930 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.528931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.528932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.528997 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.529270 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.529271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.529316 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.529399 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.529498 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:23:29.529975 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:29.530324 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.530224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52556)
0620 14:23:29.530273 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.530299 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:23:29.530718 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.530604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52566)
0620 14:23:29.530652 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.530686 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:23:29.530836 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42119: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
W20260620 14:23:29.552029 23101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:23:29.552230 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:23:29.552248 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:23:29.552296 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:23:29.553424 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:23:29.553465 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:23:29.553479 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:23:29.553493 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:23:29.553555 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:23:29.555013 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35629,127.0.98.61:43427,127.0.98.60:42119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:23:29.555265 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:23:29.555496 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:23:29.560130 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:23:29.560165 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:29.560319 23101 server_base.cc:1034] running on GCE node
W20260620 14:23:29.560165 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:23:29.560645 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:23:29.560951 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:23:29.561960 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } attempt: 1
I20260620 14:23:29.562143 23101 hybrid_clock.cc:648] HybridClock initialized: now 1781965409562135 us; error 35 us; skew 500 ppm
I20260620 14:23:29.562361 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.562223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52580)
0620 14:23:29.562273 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.562321 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:23:29.562687 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.562584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52592)
0620 14:23:29.562637 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.562666 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:23:29.562783 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42119: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:23:29.564796 23101 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:23:29.565213 23101 webserver.cc:460] Webserver started at http://127.0.98.60:33363/ using document root <none> and password file <none>
I20260620 14:23:29.565429 23101 fs_manager.cc:359] Metadata directory not provided
I20260620 14:23:29.565472 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:23:29.565624 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:23:29.566673 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data/instance:
uuid: "0a982a8dae4041b693d05a54e9a5184b"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "bb89c1c992f7cbb46b7b426bf398fbba"
server_key_iv: "4144b895b852a8e9570340ebc3105a48"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.567030 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/wal/instance:
uuid: "0a982a8dae4041b693d05a54e9a5184b"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "bb89c1c992f7cbb46b7b426bf398fbba"
server_key_iv: "4144b895b852a8e9570340ebc3105a48"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.568399 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:23:29.569099 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:29.569285 23101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:23:29.569375 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/wal
uuid: "0a982a8dae4041b693d05a54e9a5184b"
format_stamp: "Formatted at 2026-06-20 14:23:29 on dist-test-slave-h6n5"
server_key: "bb89c1c992f7cbb46b7b426bf398fbba"
server_key_iv: "4144b895b852a8e9570340ebc3105a48"
server_key_version: "encryptionkey@0"
I20260620 14:23:29.569445 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:23:29.583416 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:23:29.585027 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } attempt: 2
I20260620 14:23:29.585376 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:23:29.585434 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.585286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52608)
0620 14:23:29.585355 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.585395 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:23:29.585536 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:23:29.585842 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.585736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52614)
0620 14:23:29.585788 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.585819 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:23:29.585942 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42119: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42119: connect: Connection refused (error 111)
I20260620 14:23:29.589617 23101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42119
I20260620 14:23:29.589649 23168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42119 every 8 connection(s)
I20260620 14:23:29.590006 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:23:29.591029 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:23:29.592593 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:29.594905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTTnugX/build/release/bin/kudu as pid 23101
I20260620 14:23:29.595042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTTnugX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965139717380-392-0/minicluster-data/master-2/wal/instance
I20260620 14:23:29.595259 392 external_mini_cluster.cc:1428] Setting key 91a3ebe3b8dde19e41516841d9b2d190
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:23:29.601893 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.593283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.0.98.62:35629)
0620 14:23:29.593361 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.593364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.593543 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:23:29.594039 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.594041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.594053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.594110 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.594716 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.595275 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.596574 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.596784 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.596869 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.598998 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.599000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.599000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.599008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.600436 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.600861 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.600862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.600863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.600924 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.601138 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.601138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.601139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.601277 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.601344 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.601696 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.601774 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:23:29.601964 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.593252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35629 (local address 127.0.0.1:55598)
0620 14:23:29.593423 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.593467 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.593929 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.594189 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.594193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.594443 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.594624 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.594627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.595411 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.595413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.596483 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.596485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.596898 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.596899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.597013 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.597530 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.597538 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.598908 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.600513 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.600520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.600522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.600786 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.600996 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.600996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.600996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.601089 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.601349 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.601355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.601519 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.601644 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.601781 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260620 14:23:29.602515 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:29.604161 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.595646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.0.98.60:42119)
0620 14:23:29.595824 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.595829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.595864 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:23:29.595923 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.595925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.595944 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.596019 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.596621 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.597303 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.598038 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.598339 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.598428 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.601078 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.601086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.601086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.601092 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.603000 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.603341 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.603344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.603345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.603398 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.603623 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.603624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.603625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.603762 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.603924 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.603979 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.604055 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:23:29.609131 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.602749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43427 (local address 127.0.0.1:34462)
0620 14:23:29.602793 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.602803 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.602836 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.603009 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.603010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.603141 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.603166 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.603167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.603843 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.603844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.604464 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.604466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.604993 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.604994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.605035 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.605477 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.605482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.606528 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.608157 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.608158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.608158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.608366 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.608560 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.608561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.608562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.608627 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.608832 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.608833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.608878 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.608948 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.609034 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:23:29.609133 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.602829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.98.61:43427)
0620 14:23:29.602895 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.602897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.602904 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:23:29.602927 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.602928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.602937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.603007 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.603248 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.603768 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.604543 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.604875 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.604938 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.606617 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.606618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.606618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.606625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.608055 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.608441 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.608442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.608443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.608491 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.608687 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.608688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.608688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.608773 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.608819 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.608978 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.609048 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:23:29.609568 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:23:29.611963 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.604893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.98.62:35629)
0620 14:23:29.604929 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.604931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.604954 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:23:29.604973 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.604974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.604981 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.605038 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.605597 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.606034 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.606741 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.606938 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.607006 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.609462 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.609464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.609464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.609471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.611014 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.611312 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.611313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.611314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.611370 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.611502 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.611502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.611503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.611635 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.611788 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.611850 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.611905 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:23:29.616403 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.609797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52632)
0620 14:23:29.610014 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.610023 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.610091 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.610206 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.610207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.610358 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.610391 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.610393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.610953 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.610955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.611686 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.611688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.612147 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.612148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.612207 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.612594 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.612599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.613709 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.615406 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.615406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.615407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.615616 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.615800 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.615801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.615801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.615853 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.616092 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.616093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.616129 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.616196 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.616288 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 14:23:29.616639 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.609971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.0.98.60:42119)
0620 14:23:29.610023 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.610025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.610060 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:23:29.610143 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.610143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.610151 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.610189 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.610450 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.610879 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.611754 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.612063 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.612144 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.613806 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.613807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.613807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.613814 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.615318 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.615680 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.615681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.615681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.615729 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.615923 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.615924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.615925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.616031 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.616100 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.616222 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.616334 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":37}
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:23:29.618289 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: Bootstrap starting.
I20260620 14:23:29.619156 23169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:29.619498 23169 log.cc:824] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:29.620252 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.612493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.0.98.61:43427)
0620 14:23:29.612532 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.612534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.612545 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:23:29.612614 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.612615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.612624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.612677 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.613213 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.613702 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.614722 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.615048 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.615122 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.617449 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.617450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.617450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.617457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.618949 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.619369 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.619370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.619370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.619420 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.619710 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.619711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.619711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.619840 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.619997 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.620040 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.620131 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:23:29.620406 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: No bootstrap required, opened a new log
I20260620 14:23:29.621779 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:23:29.622378 23169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.622545 23169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a982a8dae4041b693d05a54e9a5184b, State: Initialized, Role: FOLLOWER
I20260620 14:23:29.622639 23169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.622946 23169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:29.622928 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.623057 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:29.622895 22993 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a03dd28350f34f20a67c73740ba5e037"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55602
W20260620 14:23:29.624694 23186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:29.624743 23186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:29.624969 23101 master_runner.cc:186] Error getting master registration for 127.0.98.62:35629: OK, instance_id { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" instance_seqno: 1781965409291331 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:23:29.627790 23101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:23:29.637928 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } attempt: 2
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:23:29.645844 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.638230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52638)
0620 14:23:29.638274 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.638307 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.638357 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.638482 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.638482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.638675 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.638727 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.638728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.639645 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.639646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.640430 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.640431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.640907 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.640908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.640962 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.641459 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.641464 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.642923 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.644750 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.644751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.644751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.644932 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.645141 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.645142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.645142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.645215 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.645525 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.645526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.645577 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.645648 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.645740 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 14:23:29.645857 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.638312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.0.98.60:42119)
0620 14:23:29.638354 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.638356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.638384 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:23:29.638420 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.638420 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.638431 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.638479 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.638799 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.639544 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.640514 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.640828 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.640900 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.643009 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.643011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.643011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.643018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.644674 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.645002 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.645003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.645003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.645064 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.645279 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.645280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.645280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.645453 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.645531 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.645686 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.645764 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:23:29.647636 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037: Bootstrap starting.
I20260620 14:23:29.648361 23019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:29.648679 23019 log.cc:824] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:29.649555 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037: No bootstrap required, opened a new log
I20260620 14:23:29.650913 23019 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.651054 23019 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03dd28350f34f20a67c73740ba5e037, State: Initialized, Role: FOLLOWER
I20260620 14:23:29.651140 23019 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.651468 23019 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:29.651422 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.651573 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:29.651611 22993 raft_consensus.cc:482] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:23:29.651703 22993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:29.652431 22993 raft_consensus.cc:504] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.653187 22993 leader_election.cc:288] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [CANDIDATE]: Term 1 election: Requested vote from peers 7d014893637d4d819854a01499d1c369 (127.0.98.61:43427), 0a982a8dae4041b693d05a54e9a5184b (127.0.98.60:42119)
I20260620 14:23:29.653270 23143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a03dd28350f34f20a67c73740ba5e037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0a982a8dae4041b693d05a54e9a5184b"
I20260620 14:23:29.653389 23143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:23:29.654572 23143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03dd28350f34f20a67c73740ba5e037 in term 1.
I20260620 14:23:29.654847 22968 leader_election.cc:302] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a982a8dae4041b693d05a54e9a5184b, a03dd28350f34f20a67c73740ba5e037; no voters:
I20260620 14:23:29.654984 23189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:23:29.655118 23189 raft_consensus.cc:686] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [term 1 LEADER]: Becoming Leader. State: Replica: a03dd28350f34f20a67c73740ba5e037, State: Running, Role: LEADER
I20260620 14:23:29.655197 23189 consensus_queue.cc:237] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.655629 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03dd28350f34f20a67c73740ba5e037. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.655690 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:29.654152 22951 master_runner.cc:186] Error getting master registration for 127.0.98.61:43427: OK, instance_id { permanent_uuid: "7d014893637d4d819854a01499d1c369" instance_seqno: 1781965409441305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:23:29.657311 22951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:23:29.657719 23200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:29.657804 23200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:29.664135 23211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:23:29.664347 23211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:23:29.665022 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.654768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.0.98.61:43427)
0620 14:23:29.654826 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.654828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.656774 (+ 1946us) server_negotiation.cc:407] Connection header received
0620 14:23:29.656865 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.656866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.656876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.656903 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.657642 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.658334 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.659347 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.660133 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.660176 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.661667 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.661668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.661669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.661675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.663513 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.664098 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.664099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.664100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.664168 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.664385 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.664386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.664386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.664527 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.664612 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.664839 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.664920 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:23:29.665107 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.654798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.98.62:35629)
0620 14:23:29.654962 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.654964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.655619 (+ 655us) server_negotiation.cc:407] Connection header received
0620 14:23:29.655769 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.655770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.655782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.655823 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.656619 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.657351 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.658894 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.659194 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.659235 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.661603 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.661604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.661605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.661612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.663242 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.663983 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.663985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.663985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.664054 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.664389 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.664389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.664390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.664512 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.664606 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.664924 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.665006 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 14:23:29.667233 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.655219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.0.98.60:42119)
0620 14:23:29.655252 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.655254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.657984 (+ 2730us) server_negotiation.cc:407] Connection header received
0620 14:23:29.658080 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.658081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.658089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.658117 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.658837 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.659526 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.661431 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.661727 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.661766 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.664088 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.664090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.664090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.664097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.666173 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.666445 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.666446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.666447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.666508 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.666727 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.666728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.666728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.666848 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.666932 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.667059 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.667143 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:23:29.676437 23143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [term 1 FOLLOWER]: Refusing update from remote peer a03dd28350f34f20a67c73740ba5e037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:29.676781 23208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:29.677079 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } attempt: 3
I20260620 14:23:29.678169 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03dd28350f34f20a67c73740ba5e037. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.678248 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:29.681648 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.681751 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:23:29 dist-test-slave-h6n5 krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:23:29.682271 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.682351 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:29.683004 23213 mvcc.cc:205] Tried to move back new op lower bound from 7298930318032285696 to 7298930317948416000. Current Snapshot: MvccSnapshot[applied={T|T < 7298930318032285696}]
I20260620 14:23:29.685559 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.677331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42119 (local address 127.0.0.1:52664)
0620 14:23:29.677400 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:23:29.677411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:23:29.677449 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:23:29.677677 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:23:29.677678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:23:29.677848 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:23:29.677893 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.677895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.678838 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.678840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.679692 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:23:29.679695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.680179 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.680180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:23:29.680231 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.680723 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:23:29.680728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:23:29.682238 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:23:29.684235 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.684237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.684237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.684441 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.684749 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:23:29.684750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:23:29.684750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:23:29.684816 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:23:29.685237 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:23:29.685238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:23:29.685286 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:23:29.685357 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:23:29.685456 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:23:29.685913 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:29.677514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.0.98.60:42119)
0620 14:23:29.677547 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.677549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.677556 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:23:29.677585 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.677586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.677595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.677639 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.677969 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.678739 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.679784 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:29.680071 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:29.680129 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:29.682333 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:23:29.682334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:23:29.682335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:23:29.682342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:23:29.684145 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.684523 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.684524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.684525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.684654 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:23:29.684883 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:23:29.684884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:23:29.684884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:23:29.685112 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:23:29.685244 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:29.685747 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 14:23:29.685823 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:23:29.687086 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: Bootstrap starting.
I20260620 14:23:29.687749 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: Neither blocks nor log segments found. Creating new log.
I20260620 14:23:29.688058 23211 catalog_manager.cc:1347] Generated new cluster ID: 32078f8743d342ac8b9aabdf4d032d21
I20260620 14:23:29.688055 23094 log.cc:824] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: Log is configured to *not* fsync() on all Append() calls
I20260620 14:23:29.688119 23211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:23:29.688980 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: No bootstrap required, opened a new log
I20260620 14:23:29.690960 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.691120 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d014893637d4d819854a01499d1c369, State: Initialized, Role: FOLLOWER
I20260620 14:23:29.691205 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } }
I20260620 14:23:29.691519 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:23:29.692171 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:23:29.692854 23047 tablet.cc:2369] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:23:29.693022 23067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [term 1 FOLLOWER]: Refusing update from remote peer a03dd28350f34f20a67c73740ba5e037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:23:29.693041 23068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a03dd28350f34f20a67c73740ba5e037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d014893637d4d819854a01499d1c369"
I20260620 14:23:29.693401 23216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:23:29.693809 23068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03dd28350f34f20a67c73740ba5e037 in term 1.
I20260620 14:23:29.694854 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.694991 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:29.695530 23231 mvcc.cc:205] Tried to move back new op lower bound from 7298930318032285696 to 7298930317948416000. Current Snapshot: MvccSnapshot[applied={T|T < 7298930318032285696}]
I20260620 14:23:29.695704 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.695837 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:23:29.696589 23234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:23:29.696647 23234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:23:29.695086 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03dd28350f34f20a67c73740ba5e037. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.703444 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a03dd28350f34f20a67c73740ba5e037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a03dd28350f34f20a67c73740ba5e037" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35629 } } peers { permanent_uuid: "7d014893637d4d819854a01499d1c369" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43427 } } peers { permanent_uuid: "0a982a8dae4041b693d05a54e9a5184b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42119 } } }
I20260620 14:23:29.703996 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037 [sys.catalog]: This master's current role is: LEADER
I20260620 14:23:29.704048 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:23:29.705340 23211 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:23:29.706053 23211 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:23:29.716032 23211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a03dd28350f34f20a67c73740ba5e037: Generated new TSK 0
I20260620 14:23:29.716226 23211 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:23:29.743060 23023 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.741212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:35629)
0620 14:23:29.741258 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.741260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.742371 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 14:23:29.742481 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.742482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.742493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.742524 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.742990 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55614: BlockingRecv error: recv got EOF from 127.0.0.1:55614 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 14:23:29.743858 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.741640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.0.98.61:43427)
0620 14:23:29.741680 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.741682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.743312 (+ 1630us) server_negotiation.cc:407] Connection header received
0620 14:23:29.743411 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.743412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.743422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.743453 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.743783 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34492: BlockingRecv error: recv got EOF from 127.0.0.1:34492 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:23:29.744656 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.741943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.0.98.60:42119)
0620 14:23:29.741981 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.741983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.744071 (+ 2088us) server_negotiation.cc:407] Connection header received
0620 14:23:29.744234 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.744235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.744246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.744274 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.744597 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52666: BlockingRecv error: recv got EOF from 127.0.0.1:52666 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:23:29.746943 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.746210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.0.98.61:43427)
0620 14:23:29.746252 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.746254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.746442 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:23:29.746545 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.746546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.746563 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.746591 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.746886 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34496: BlockingRecv error: recv got EOF from 127.0.0.1:34496 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:23:29.749938 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.749214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.0.98.61:43427)
0620 14:23:29.749247 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.749249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.749469 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:23:29.749562 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.749562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.749572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.749604 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.749875 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34510: BlockingRecv error: recv got EOF from 127.0.0.1:34510 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:23:29.753868 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.753129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.0.98.61:43427)
0620 14:23:29.753166 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.753168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.753381 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:23:29.753473 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.753474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.753483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.753511 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.753824 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34514: BlockingRecv error: recv got EOF from 127.0.0.1:34514 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:23:29.759770 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.758986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.0.98.61:43427)
0620 14:23:29.759020 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.759021 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.759237 (+ 216us) server_negotiation.cc:407] Connection header received
0620 14:23:29.759331 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.759332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.759342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.759370 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.759721 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34518: BlockingRecv error: recv got EOF from 127.0.0.1:34518 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:23:29.769834 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.768987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.0.98.61:43427)
0620 14:23:29.769026 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.769028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.769270 (+ 242us) server_negotiation.cc:407] Connection header received
0620 14:23:29.769372 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.769374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.769384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.769415 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.769766 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34524: BlockingRecv error: recv got EOF from 127.0.0.1:34524 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:23:29.788183 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.787179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.98.61:43427)
0620 14:23:29.787223 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.787225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.787501 (+ 276us) server_negotiation.cc:407] Connection header received
0620 14:23:29.787663 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.787664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.787676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.787706 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.788125 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34534: BlockingRecv error: recv got EOF from 127.0.0.1:34534 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:23:29.823480 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.822354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.0.98.61:43427)
0620 14:23:29.822407 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.822409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.822719 (+ 310us) server_negotiation.cc:407] Connection header received
0620 14:23:29.822871 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.822872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.822883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.822920 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.823420 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34546: BlockingRecv error: recv got EOF from 127.0.0.1:34546 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:23:29.889823 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:29.888992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.0.98.61:43427)
0620 14:23:29.889106 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:29.889108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:29.889157 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:23:29.889215 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:29.889216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:29.889227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:29.889270 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:29.889727 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34560: BlockingRecv error: recv got EOF from 127.0.0.1:34560 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 14:23:30.019745 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:30.019033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.0.98.61:43427)
0620 14:23:30.019140 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:30.019142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:30.019157 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:23:30.019234 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:30.019235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:30.019245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:30.019312 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:30.019654 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34570: BlockingRecv error: recv got EOF from 127.0.0.1:34570 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 14:23:30.277634 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:23:30.277000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.61:43427)
0620 14:23:30.277079 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:30.277081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:30.277092 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:23:30.277148 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:30.277149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:30.277159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:30.277224 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:23:30.277588 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34574: BlockingRecv error: recv got EOF from 127.0.0.1:34574 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:23:30.625173 23186 catalog_manager.cc:1259] Loaded cluster ID: 32078f8743d342ac8b9aabdf4d032d21
I20260620 14:23:30.625252 23186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: loading cluster ID for follower catalog manager: success
I20260620 14:23:30.626107 23186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: acquiring CA information for follower catalog manager: success
I20260620 14:23:30.626426 23186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a982a8dae4041b693d05a54e9a5184b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:23:30.697065 23234 catalog_manager.cc:1259] Loaded cluster ID: 32078f8743d342ac8b9aabdf4d032d21
I20260620 14:23:30.697144 23234 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: loading cluster ID for follower catalog manager: success
I20260620 14:23:30.697990 23234 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: acquiring CA information for follower catalog manager: success
I20260620 14:23:30.698314 23234 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d014893637d4d819854a01499d1c369: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:23:30.793839 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:30.790876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.98.61:43427)
0620 14:23:30.791017 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:30.791019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:30.791044 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:23:30.791119 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:30.791120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:30.791170 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:30.791256 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:30.791497 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.792291 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.793142 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.793341 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.793400 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:30.793549 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:30.793658 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:30.793718 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:30.793756 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:23:30.793774 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260620 14:23:30.797952 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:30.795706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.0.98.60:42119)
0620 14:23:30.795824 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:30.795826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:30.795834 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:23:30.795892 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:30.795893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:30.795926 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:30.795992 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:30.796089 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.796575 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.797299 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.797481 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.797543 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:30.797673 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:30.797768 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:30.797824 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:30.797875 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:23:30.797890 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260620 14:23:30.802657 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:30.799920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.98.60:42119)
0620 14:23:30.799961 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:30.799963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:30.799988 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:23:30.800045 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:30.800045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:30.800056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:30.800108 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:30.800275 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.800734 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.801778 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.802101 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.802158 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:30.802356 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:30.802394 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:30.802459 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:30.802539 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:23:30.802561 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:23:30.802805 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:30.799632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.0.98.62:35629)
0620 14:23:30.799755 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:30.799757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:30.799767 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:23:30.799811 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:30.799812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:30.799865 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:30.800297 (+ 432us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:30.800339 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.801082 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.801823 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.802087 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.802150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:30.802345 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:30.802455 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:30.802524 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:30.802603 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:23:30.802625 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260620 14:23:30.802937 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:23:30.799956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.98.61:43427)
0620 14:23:30.800012 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:23:30.800014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:23:30.800022 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:23:30.800045 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:23:30.800046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:23:30.800056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:23:30.800108 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:23:30.800245 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.800999 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.802027 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:23:30.802303 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:23:30.802364 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:23:30.802651 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:23:30.802686 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:23:30.802715 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 14:23:30.802822 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:23:30.802848 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:23:30.809991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 22951
I20260620 14:23:30.815362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 23026
I20260620 14:23:30.820221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTTnugX/build/release/bin/kudu with pid 23101
2026-06-20T14:23:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2807 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5588 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271108 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST